which function of gdb overrides the exception handler to handle the "int 3" exception?
it is the scenario of SW breakpoint where instruction is replaced by int 3 and gdb takes the control.
you want to debug the debugger? or create a custom debugger with your own breakpoint handler? as a user space program runnable by non-root users gdb certainly has no means to directly catch the interrupt; it has to rely on the kernel. have you studied the ptrace() syscall?
yes, I have studied ptrace system call. but ptrace put the child to sleep. that is what I do not want.
I want to just create custom debugger. which will just trace the running process at particular tracepoint.
that means :
1) attach the process.
2) process does not stop.
3) process is running
4) gdb (custom debugger) is also running
5) you can set trace point now..
6) when trace point is hit just collect debug information....(but the debugee should not be put to sleep or affected by schedular. just let debugger collect the information and control is back to gdb. no scheduling overhead of kernel such as giving signal continue to child process if we use ptrace.
I hope it is clear.
this looks more like data collection than it looks like a job for a debugger. at least others have solved similar problems. looked at OProfile, SystemTap, Frysk, LTT, etc. ?
have you measured the ptrace overhead? sampling data has overhead at any case. using ptrace has the advantages to freeze the child process while you sample the otherwise volatile data and that you don't have to transport your sampling data from some in-kernel-buffer to userspace, since the sampling takes place in userspace. you could also start a second thread in your debugee with the sole purpose to sample the other thread's data. as the data is unstable anyway (since you want your process to run while sampling) you can just sample the whole time.
yah, I looked at LTT and oprofile.
what you said is correct : the overhead will always be there.
but one important statement about ptrace you told is : ptrace makes running process freeze.
that is what I do not want....
I have two aproaches.
1) let kernel module collect data and when exception (sw breakpoint int3) occurrs. and control return to the process once exception is over. in that case we do not use ptrace.
but I think everything has to re written from scratch and also in the kernel space.
its lots of effort.
2) use ptrace, but as ptrace make the child process stop....modify debugger in such a way that at the end of exception give a signal to the child to continue the process because it is just a trace point (not breakpoint) so process should not stop.
another thing is the control (terminal) should always be with gdb while process is running. so I think another thread of execution we need to create.
in above approach, only overhead is scheduling. menas when child stops when trace point is hit, it is specifically made to go to sleep by ptrace. which I do not want.
I hope scenario is clear now.
forgot to mention very important point :
the debugger should be source level debugger.
should not be like oprofile, LTT, frysk.