Jun 01, 2021 Article blog
In this article, we'll show you
how application and system engineers comb through Linux kernel code from the perspective of non-kernel developers.
I hope you can get something after reading, but also hope that more developers can pay attention to the field of kernel development, after all, even the ancestral master
Linus
said that the kernel maintainer to follow up no one ah!
Test environment version information:
Ubuntu(lsb_release -a) | Distributor ID: UbuntuDescription: Ubuntu 19.10Release: 19.10 |
---|---|
Linux(uname -a) | Linux yahua 5.5.5 #1 SMP ... x86_64 x86_64 x86_64 GNU/Linux |
Java | Openjdk jdk14 |
How do people who play the kernel know
Java
This is mainly due to my school's
Java
course and graduation experience of doing
Android
phones at Huawei, several modules scanned from
APP/Framework/Service/HAL/Driver
and naturally have an understanding of
Java
Every time I mention
Java
I think of an interesting experience. J
ust after graduation to the department to report to the first week, the department leader (in Huawei is the Manager) arranged for us to familiarize ourselves with
Android
I
t took me a few days to write an
Android
game, some of which was similar to watching it all the time.
At the beginning of the week, the leader saw my presentation and looked unhappy, questioning that my direct leader (called PL, Project Leader at Huawei) had not told us the direction of the department.
emm, I really didn't understand what the so-called familiar
Android
was supposed to do, and then PL said it was about getting familiar with the xxx module, and the APP was just one part of it.
So if I'm sure, maybe I'm a
Java
engineer now (haha manual dog's head).
(Recommended tutorial: Java tutorial)
The furthest distance in the world is when we're sitting next door, I'm looking at the underlying agreement, and you're studying spring...
If you want to get closer to us, download
openjdk
source code, then
glibc
and then
kernel.
内核源码
Java
program to
JVM
this is certainly more familiar to me, on the door to make an axe.
Let's take the entry of
JVM
as an example and analyze the process from
JVM
to the kernel, which is
main
function (java.base/share/native/launcher/main.c):
JNIEXPORT int
main(int argc, char **argv)
{
//中间省略一万行参数处理代码
return JLI_Launch(margc, margv,
jargc, (const char**) jargv,
0, NULL,
VERSION_STRING,
DOT_VERSION,
(const_progname != NULL) ? const_progname : *margv,
(const_launcher != NULL) ? const_launcher : *margv,
jargc > 0,
const_cpwildcard, const_javaw, 0);
}
JLI_Launch
did three things we cared about.
First, call
CreateExecutionEnvironment
to find the settings environment variables, such as the path of
JVM
(the variable
jvmpath
below), and in my case,
/usr/lib/jvm/java-14-openjdk-amd64/lib/server/libjvm.so
window
platform may be
libjvm.dll
Second, call
LoadJavaVM
to load
JVM
which is
libjvm.so
the file, and then find the corresponding field for the function assignment that created the
JVM
to
InvocationFunctions
jboolean LoadJavaVM(const char *jvmpath, InvocationFunctions *ifn)
{
void *libjvm;
//省略出错处理
libjvm = dlopen(jvmpath, RTLD_NOW + RTLD_GLOBAL);
ifn->CreateJavaVM = (CreateJavaVM_t)
dlsym(libjvm, "JNI_CreateJavaVM");
ifn->GetDefaultJavaVMInitArgs = (GetDefaultJavaVMInitArgs_t)
dlsym(libjvm, "JNI_GetDefaultJavaVMInitArgs");
ifn->GetCreatedJavaVMs = (GetCreatedJavaVMs_t)
dlsym(libjvm, "JNI_GetCreatedJavaVMs");
return JNI_TRUE;
}
dlopen
and
dlsym
involve dynamic links, which are simply understood
libjvm.so
definitions that contain
JNI_CreateJavaVM
JNI_GetDefaultJavaVMInitArgs
and
JNI_GetCreatedJavaVMs
and when dynamic links are completed,
ifn->CreateJavaVM
ifn->GetDefaultJavaVMInitArgs
and
ifn->GetCreatedJavaVMs
are the addresses of these functions.
You may want to confirm that the next
libjvm.so
have these three functions.
objdump -D /usr/lib/jvm/java-14-openjdk-amd64/lib/server/libjvm.so | grep -E
"CreateJavaVM|GetDefaultJavaVMInitArgs|GetCreatedJavaVMs" | grep ":$"
00000000008fa9d0 <JNI_GetDefaultJavaVMInitArgs@@SUNWprivate_1.1>:
00000000008faa20 <JNI_GetCreatedJavaVMs@@SUNWprivate_1.1>:
00000000009098e0 <JNI_CreateJavaVM@@SUNWprivate_1.1>:
These implementations are available in
openjdk
source code (hotspot/share/prims/under), and interested students can continue to delve into them.
Finally, call
JVMInit
initialize
JVM
load Java
program.
JVMInit
ContinueInNewThread
which
CallJavaMainInNewThread
T
o paraphernate, I really don't like to tell the story the way function calls, a call b, b and c, is a waste of space, but in some places the span is too large for fear of misunderstanding (especially for beginners).
Believe me, water injection, really no, I don't need experience haha.
The main logic of
CallJavaMainInNewThread
is as follows:
int CallJavaMainInNewThread(jlong stack_size, void* args) {
int rslt;
pthread_t tid;
pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);
if (stack_size > 0) {
pthread_attr_setstacksize(&attr, stack_size);
}
pthread_attr_setguardsize(&attr, 0); // no pthread guard page on java threads
if (pthread_create(&tid, &attr, ThreadJavaMain, args) == 0) {
void* tmp;
pthread_join(tid, &tmp);
rslt = (int)(intptr_t)tmp;
}
else {
rslt = JavaMain(args);
}
pthread_attr_destroy(&attr);
return rslt;
}
See
pthread_create
solve the case,
Java
thread is implemented through
pthread
Y
ou can get into the kernel here, but let's move on to
JVM
first.
ThreadJavaMain
calls
JavaMain
directly, so the logic here is that if the thread is created successfully,
JavaMain
is executed by a new thread, otherwise
JavaMain
is executed in the current process.
JavaMain
is our focus, and the core logic is as follows:
int JavaMain(void* _args)
{
JavaMainArgs *args = (JavaMainArgs *)_args;
int argc = args->argc;
char **argv = args->argv;
int mode = args->mode;
char *what = args->what;
InvocationFunctions ifn = args->ifn;
JavaVM *vm = 0;
JNIEnv *env = 0;
jclass mainClass = NULL;
jclass appClass = NULL; // actual application class being launched
jmethodID mainID;
jobjectArray mainArgs;
int ret = 0;
jlong start, end;
/* Initialize the virtual machine */
if (!InitializeJVM(&vm, &env, &ifn)) { //1
JLI_ReportErrorMessage(JVM_ERROR1);
exit(1);
}
mainClass = LoadMainClass(env, mode, what); //2
CHECK_EXCEPTION_NULL_LEAVE(mainClass);
mainArgs = CreateApplicationArgs(env, argv, argc);
CHECK_EXCEPTION_NULL_LEAVE(mainArgs);
mainID = (*env)->GetStaticMethodID(env, mainClass, "main",
"([Ljava/lang/String;)V"); //3
CHECK_EXCEPTION_NULL_LEAVE(mainID);
/* Invoke main method. */
(*env)->CallStaticVoidMethod(env, mainClass, mainID, mainArgs); //4
ret = (*env)->ExceptionOccurred(env) == NULL ? 0 : 1;
LEAVE();
}
Step 1, call
InitializeJVM
initialize
JVM
InitializeJVM
calls
ifn->CreateJavaVM
which is
libjvm.so
in
JNI_CreateJavaVM
Step 2,
LoadMainClass
which ends up calling
JVM_FindClassFromBootLoader
also finds the function through a dynamic link (defined under hotspot/share/prims/) and then calls it.
Step 3 and 4,
Java
classmates should know that this is calling
main
function.
It's a bit of a running point...
pthread_create
continue to look at the kernel with pthread_create as an example.
In fact,
pthread_create
is a short distance from the kernel, which is
glibc
nptl/pthread_create.c
The creation thread is ultimately implemented through
clone
system call, and we don't care about the details of
glibc
(otherwise it's off again), just look at how it differs from the direct
clone
(Recommended micro-class: Java micro-class)
The following discussion of threads is extracted from the book.
const int clone_flags = (CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SYSVSEM
| CLONE_SIGHAND | CLONE_THREAD
| CLONE_SETTLS | CLONE_PARENT_SETTID
| CLONE_CHILD_CLEARTID
| 0);
__clone (&start_thread, stackaddr, clone_flags, pd, &pd->tid, tp, &pd->tid);
The description of each sign is as follows (this sentence is not excerpted... )。
sign | description |
---|---|
CLONE_VM | Share the VM with the current process |
CLONE_FS | Share file system information |
CLONE_FILES | Share open files |
CLONE_PARENT | The same parent process as the current process |
CLONE_THREAD | Being in the same thread group as the current process also means that the thread is created |
CLONE_SYSVSEM | Share sem_undo_list |
...... | ...... |
Share VMs with current processes, share file system information, share open files... We understand when we see this, and that's what threading is all about.
Linux
doesn't actually essentially separate processes from threads, also known as lightweight processes (Low Weight Process, LWP), except that threads share memory, files, and other resources with the process (thread) that created it.
The full paragraph is as follows (several paragraphs expanded in double quotes) that interested students can read in detail:
The
_do_fork
parameter that
fork
passes to
clone_flags
is fixed, so it can only be used to create a process, the kernel provides another system call
clone
and
clone
eventually calls
_do_fork
implementation, unlike
fork
where the user can determine
clone_flags
as needed, and we can use it to create threads, as follows (the parameters of
clone
may be different under different platforms):
SYSCALL_DEFINE5(clone, unsigned long, clone_flags, unsigned long, newsp,
int __user *, parent_tidptr, int, tls_val, int __user *, child_tidptr)
{
return _do_fork(clone_flags, newsp, 0, parent_tidptr, child_tidptr);
}
Linux
treats threads as lightweight processes, but the characteristics of threads are not arbitrarily determined by
Linux
and should be as compatible as possible with other operating systems, so it
POSIX
standard requirements for threads.
Therefore, to create a thread, the parameters passed to the
clone
system call should also be basically fixed.
The parameters for creating a thread are complex, and fortunately
pthread
thread provides us with a function that
pthread_create
and the function prototype (user space) is as follows.
int pthread_create(pthread_t *thread, const pthread_attr_t *attr,
void *(*start_routine) (void *), void *arg);
The first argument
thread
is an output parameter in which the thread's id is
id
after the thread is created successfully, and the second parameter is used to customize the properties of the new thread.
The successful creation of a new thread executes a function that
start_routine
points to, and the argument passed to that function is
arg
pthread_create
how exactly
clone
is called, roughly as follows:
//来源: glibc
const int clone_flags = (CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SYSVSEM
| CLONE_SIGHAND | CLONE_THREAD
| CLONE_SETTLS | CLONE_PARENT_SETTID
| CLONE_CHILD_CLEARTID
| 0);
__clone (&start_thread, stackaddr, clone_flags, pd, &pd->tid, tp, &pd->tid);
clone_flags
have more flags for positioning, the first few flags indicate that the thread shares resources with the current process (and possibly the thread),
CLONE_THREAD
means that the new thread and the current process are not parent-child relationships.
clone
system call is also ultimately implemented through
_do_fork
so the difference between it and
fork
of the creation process is limited to differences due to different parameters, and there are two questions to explain.
First,
vfork
places
CLONE_VM
flag, causing the new process to modify the local variable to affect the current process. S
o the
clone
which is also placed
CLONE_VM
also has this hidden danger? T
he answer is no, because the new thread specifies its own user stack, specified by
stackaddr
copy_thread
the
sp
parameter of the function is
stackaddr
childregs->sp = sp
modifies the pt_regs of
pt_regs
new thread, so that when the new thread executes in user space, it uses a different stack than the current process and does not cause interference.
Then why
vfork
do this, please refer to
vfork
design intent.
Second,
fork
returns twice, and so does
clone
but they are all returned to the system call and start executing,
pthread_create
How do I get a new thread to execute
start_routine
start_routine
is performed indirectly by
start_thread
function, so we just need to understand how
start_thread
is called.
start_thread
is not passed to
clone
system call, so its call is independent of the kernel, and the answer is
__clone
function.
(Recommended tutorial: Linux tutorial)
In order to fully understand how the new process uses its user stack and the calling procedure of
start_thread
it is necessary to analyze
__clone
function, even if it is platform-dependent and written in assembly language.
/*i386*/
ENTRY (__clone)
movl $-EINVAL,%eax
movl FUNC(%esp),%ecx /* no NULL function pointers */
testl %ecx,%ecx
jz SYSCALL_ERROR_LABEL
movl STACK(%esp),%ecx /* no NULL stack pointers */ //1
testl %ecx,%ecx
jz SYSCALL_ERROR_LABEL
andl $0xfffffff0, %ecx /*对齐*/ //2
subl $28,%ecx
movl ARG(%esp),%eax /* no negative argument counts */
movl %eax,12(%ecx)
movl FUNC(%esp),%eax
movl %eax,8(%ecx)
movl $0,4(%ecx)
pushl %ebx //3
pushl %esi
pushl %edi
movl TLS+12(%esp),%esi //4
movl PTID+12(%esp),%edx
movl FLAGS+12(%esp),%ebx
movl CTID+12(%esp),%edi
movl $SYS_ify(clone),%eax
movl %ebx, (%ecx) //5
int $0x80 //6
popl %edi //7
popl %esi
popl %ebx
test %eax,%eax //8
jl SYSCALL_ERROR_LABEL
jz L(thread_start)
ret //9
L(thread_start): //10
movl %esi,%ebp /* terminate the stack frame */
testl $CLONE_VM, %edi
je L(newpid)
L(haspid):
call *%ebx
/*…*/
Take, for example,
__clone
(start_thread, stackaddr, clone_flags, pd, sptd->tid, tp, spt->tid),
FUNC(%esp)
&start_thread
to the start_thread ,
STACK(%esp)
corresponds to
stackaddr
ARG(%esp)
corresponds to
pd
(the parameter passed by the new process to
start_thread
stackaddr
the new process to
ecx
to ensure that its value is not 0.
pd
&start_thread
and 0 into the stack of the new thread, with no effect on the stack of the current process.
esp
registers by 12 accordingly.
FLAGS+12(%esp)
into
ebx
corresponding to
clone_flags
and
clone
system call number into eax.
clone_flags
into the stack of the new process.
int
directive to initiate a system call and hand it over to the kernel to create a new thread.
As of here, all code is executed by the current process, and the new thread is not executed.
ret_from_fork
and when it switches to the user state, its stack
stackaddr
so its
edi
is equal to
clone_flags
esi
is equal to 0,
ebx
is equal to
&start_thread
eax
step 8 determines the result of the
clone
system call, and for the current process, if
clone
system call succeeds in returning
id
of the new thread in its
pid namespace
greater than 0, so it
ret
exit
__clone
function. F
or a new thread, the return value of the
clone
system call is equal to 0, so it executes the code at
L(thread_start)
clone_flags
CLONE_VM
flag is set, the
call *%ebx
executed,
ebx
is equal to
&start_thread
and
start_thread
is executed, and it calls the start_routine provided to the
pthread_create
ending.
start_routine
”
In this way,
Java
→
JVM
→
glibc
→
内核
as if not far away.
(Recommended micro-class: Linux micro-class)
This is a related description of how far
Java
is from the
Linux
kernel, and I hope it will help you.