CPU scheduling and migration

[kris.c]

Hi,
I have been playing around with cpu_schedaffinity() and other related calls. I am running a parallel job on an AMD Barcelona machine (16 cores in each node). For a specific reason, I had to swap the jobs that were running on the first 8 cores with the next 8 cores, and then again swap them back. I was able to get sched_affinity to make this happen, and I also looked at the "top" output to see that the processes are indeed getting migrated to the exact core. I just want to be a 100% sure that it is happening correctly and I think the best way out is to have each process print out the processor id that it is running on. Is there any API that can give me such information.

Thanks!

[aram535]

I'm pretty sure I've seen that in the AMD codeanalyzer but I can't be 100% sure. Check it out. Maybe you can figure out what they're using.

[DandePhaniraj]

If you want to check which process is running on which core in SMP environment, you can use top command.

Enter top command on shell prompt, after top is running, press f , j and Enter keys from the keyboard.
After that you can see 'P' field in the top command which gives the last used cpu information for the process.

Ex:- PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND

[kris.c]

Thanks for the replies. I ran into some library dependency issues with the CodeAnalyzer and I havent had the time to dig around for it, yet.
I actually used the same set of commands on the top to verify that the migration was indeed happening. I am also interested in measuring the overhead of this operation. I noticed that the cost of doing a set_affinity is considerable high. Are there any system/kernel level optimizations that can be done to lower the overhead? The following is the piece of code that I was running to measure the latency. I was running it on a 16 core Barcelona machine. I am trying to swap the processes on the first 8 cores with those on the next 8 cores. So, I have set shmem_comm_size to 8.

int main( int argc, char *argv[] )
{
int rank, size;
int shmem_comm_size = 8;
double time_start, time_end;

MPI_Init(&argc,&argv);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);

cpu_set_t mask;
if (sched_getaffinity(0, sizeof(cpu_set_t), &mask) < 0) {
perror("sched_getaffinity");
return -1;
}

time_start = MPI_Wtime();
CPU_ZERO(&(mask));
CPU_SET((rank ^ shmem_comm_size),&(mask));

if (sched_setaffinity(0, sizeof(cpu_set_t), &mask) < 0) {
perror("sched_setaffinity");
return -1;
}
time_end = MPI_Wtime();

printf("%d time for first migration %f \n",rank, (time_end - time_start)*1e6);

MPI_Finalize();
return 0;
}


And these are the timing results that I got :
0 time for first migration 32.901764
1 time for first migration 33.855438
2 time for first migration 30.994415
3 time for first migration 33.855438
4 time for first migration 30.040741
5 time for first migration 99986.076355
6 time for first migration 29.087067
7 time for first migration 44.107437
8 time for first migration 116982.936859
9 time for first migration 202950.954437
10 time for first migration 116971.969604
11 time for first migration 202919.960022
12 time for first migration 4116.058350
13 time for first migration 36.001205
14 time for first migration 191.926956
15 time for first migration 30.994415