May 23, 2021 Shell - An example of programming
2. The location of the file system in the Linux operating system
3. Hardware management and device drivers
4. Understand and view disk partitions
5. The relationship between partitioning and file systems
Prepared for a long time, looking for many days of information, do not know how to write: because of fear of not being able to hold, so on the one hand continue to find information, on the other hand, think about how to write. As part of the Shell Programming Paradigm, it is hoped that it will be a good way for Shell programmers to understand how to use shell commands to perform various operations on file systems that are very signed to the Linux system, and that the "chaos" of the file system among Shell programmers will disappear, in the hope that the file system will be presented in a clearer light.
How do I get to know the file system? F rom the shell programmer's point of view, a file system is a way to organize various files. H owever, file systems cannot exist independently of hardware storage devices and operating systems, so it is necessary to understand the links between various concepts such as hardware storage devices, partitions, operating systems, logical volumes, file systems, etc., in order to understand some of the "details" of the normal operation of the file system. This connection may (and may have some problems) be presented in one way or another:
From the diagram, we can clearly see the relationship between the various "concepts", they are distributed at different levels, covering hardware devices, system kernel space, system user space. I n the user space, users can operate specific hardware devices regardless of the kernel, using only a variety of interfaces designed by the programmer, and the average programmer only needs to use the various interfaces provided by the kernel (System Call) or some C library to interact with the kernel, without having to care about the specific implementation details. For operating system developers, however, they need to design specific data structures in kernel space to manage and organize the underlying hardware devices.
Here's a bottom-up approach (starting with the underlying hardware) that uses tools to analyze and understand several important concepts in the diagram. (If you are interested, you can first look at the following information)
Resources:
Linux systems manage hardware devices through device drivers. I
f you add a new hardware device, you need to write the appropriate hardware driver to manage it. F
or some common hardware devices, the system already has its own drivers, and when you compile the kernel, select them, then compile them as part of the kernel or as modules.
If compiled as a module, the corresponding module file can be found in the
/lib/modules/$(uname -r)
directory.
For example, the corresponding scsi driver and usb driver module files can be found in this way:
Update the system Chinese index database (a little slow)
$ updatedb
Find scsi-related drivers
$ locate scsi*.ko
Look for usb-related drivers
$ locate usb*.ko
These drivers are
.ko
with .ko and are compiled as modules by default when the system is installed. Y
ou can actually compile them as part of the kernel, just by selecting it when
[*]
H
owever, in many cases they are compiled as modules, which reduces the size of the kernels and gives you the flexibility to load and unload them as needed.
Here's a quick demonstration of how to uninstall the module, load the module, and see the status of the loaded module.
The status of the individual modules loaded
modules
be checked through the modules file of the
/proc
file
lsmod
command.
$ cat /proc/modules
Or
$ lsmod
View scsi and usb-related drivers, and the results are listed as module names, module sizes, references to other modules (number of references, modules that reference them)
$ lsmod | egrep "scsi|usb"
usbhid 29536 0
hid 28928 1 usbhid
usbcore 138632 4 usbhid,ehci_hcd,ohci_hcd
scsi_mod 147084 4 sg,sr_mod,sd_mod,libata
Let's
usbhid
module to see (don't uninstall the scsi driver!)
Because your system may be running on it, if you really want to play, remember to save the
rmmod
command and switch to root user first:
$ sudo -s
# rmmod usbhid
Look at the module's information again, you can't see it enough
$ lsmod | grep ^usbhid
If there is a usb mouse, then move it, do you find that you can't move it? B
ecause the equipment drive is gone, the equipment will naturally not be able to use Luo.
But don't be nervous, now that you know why, reload the driver and reload
usbhid
below.
insmod
$ sudo -s
# insmod `locate usbhid.ko`
locate usbhid.ko
the
usbhid.ko
if there is no
updatedb
that it can not be
/lib/modules
find
usbhid.ko
file to find.
# insmod $(find /lib/modules -name "*usbhid.ko*" | grep `uname -r`)
Now the mouse can be used again, do not believe to move the mouse again : -)
Here, the relationship between hardware devices and device drivers should be clearer. If not, continue with the following.
Linux device drivers are associated with the appropriate device files, which correspond to hardware devices one by one.
These device files are stored uniformly in the
/dev/
For example, the scsi device
/dev/sda
/dev/sda1
/dev/sda2
...
Check out these device information below.
$ ls -l /dev/sda*
brw-rw---- 1 root disk 8, 0 2007-12-28 22:49 /dev/sda
brw-rw---- 1 root disk 8, 1 2007-12-28 22:50 /dev/sda1
brw-rw---- 1 root disk 8, 3 2007-12-28 22:49 /dev/sda3
brw-rw---- 1 root disk 8, 4 2007-12-28 22:49 /dev/sda4
brw-rw---- 1 root disk 8, 5 2007-12-28 22:50 /dev/sda5
brw-rw---- 1 root disk 8, 6 2007-12-28 22:50 /dev/sda6
brw-rw---- 1 root disk 8, 7 2007-12-28 22:50 /dev/sda7
brw-rw---- 1 root disk 8, 8 2007-12-28 22:50 /dev/sda8
You can see that the first character in the first column
b
and the fifth column is the number 8.
b
that the file is a block device file, correspondingly,
c
a character device (e.g. '/dev/ttyS0), you can see here the difference between the block device and the character device:
Character devices: Character devices are devices that can be accessed like byte streams, and character terminals and serials belong to character devices.
- Block device: A file system can be accommodated on a block device. U nlike character devices, block devices can only transfer one or more complete blocks at a time when reading and writing. I n the Linux operating system, applications can read and write block devices (read or write arbitrary bytes of data at once) just like accessing character devices. Therefore, the difference between block devices and character devices is simply the difference in the management of data in the kernel.
The number 8 is the device number corresponding to the hardware device in the kernel, and the device number assignment can be found in the
Documentation/devices.txt
/proc/devices
of the kernel. B
ut why does the same device correspond to different device files (why are there different numbers after
/dev/sda
and column 6 in
ls
result corresponds to them). T
his is actually to distinguish between different parts of different devices. F
or hard drives, this can handle different partitions inside the hard drive. I
n the case of the kernel, it only needs to be able to find the corresponding hardware device through the device number in column 5, but for the driver module it also needs to know how to handle different partitions, so there is an additional auxiliary device number, which is the corresponding content in column 6.
Such a device has a master device number (column 5) and an auxiliary device number (column 6), making it easy to manage a variety of hardware devices.
Because the device files and hardware correspond, the hard drive information can be read directly from
/dev/sda
(in the case
IDE
hard drive, the corresponding device is the
/dev/hda
device, for example:
Copy the first 512 bytes of the hard disk with the
dd
command to root the user
$ sudo dd if=/dev/sda of=mbr.bin bs=512 count=1
Use
file
command to view the appropriate information
$ file mbr.bin
mbr.bin: x86 boot sector, LInux i386 boot LOader; partition 3: ID=0x82, starthead 254, startsector 19535040, 1959930 sectors; partition 4: ID=0x5, starthead 254, startsector 21494970, 56661255 sectors, code offset 0x48
You can also read and analyze in a 16-step form with the
od
command
$ od -x mbr.bin
bs
the size of the block (in
bytes
and
count
number of blocks
Because this information is not intuitive (and further in-depth analysis is provided below), a look at another device file will provide a very intuitive demonstration of the relationship between the device file and the hardware. Or take the mouse as an example, the following to read the mouse corresponding to the device file information.
$ sudo -s
# cat /dev/input/mouse1 | od -x
Your mouse driver may not be the same, so the device files may be other, but they'll
/dev/input
Move the mouse to see if you find different information outputs.
Based on this principle, we often check that the serial line is corrupted by reading the contents of the device file
/dev/ttyS0
at one end and writing to the device file
/dev/ttyS0
at the other end.
Here, you should be more impressed by the correlation between device drivers, device files, and hardware devices. If you want to learn more about how device-driven works and how device-driven writing works, take a look at the information listed below to begin the process of device-driven writing.
Resources:
Memory, usb sticks, and so on can actually be used as "storage" devices at the bottom of the file system, but only hard drives are used as instances to describe the relationship between disks and partitions.
Currently, Linux's partitioning still uses the partitioning principle used by the first PC hard drive, which is analyzed and demonstrated step by step below.
Let's take a look at a few concepts:
Device management and partitioning
Under Linux, each storage device corresponds to a system's device file, and
IDE
and
SCSI
the corresponding device files containing the character hd and
sd
can be
hd
/dev
directory. D
epending on the board device interface and data cable interface of the
sd
a to
z
can be added after the
hd
or
a
characters, such as
hda
hdb
hdc
and
sda
sdb
sdc
a number can be
hda1
hda2
hda3
and
sda1
sda2
sda3, in order to distinguish between different partitions of
sda3
, so in
/dev
you can see many similar device files.
The role of each partition
The problem of primary and logical partitioning is often encountered in partitioning, which is actually to facilitate the extension of partitioning, just as the subsequent introduction of logical volumes is to better manage multiple hard drives, the introduction of primary partitioning and logical partitioning can easily be the management of partitioning.
Each hard disk device in a Linux system consists of up to 4 primary partitions, including extended partitions.
The role of the primary partition is that the computer is used to start the operating system, so the startup program of each operating system, or boot program, should be stored on the main partition. L
inux states that the primary partition (or extended partition) occupies the first four partition numbers.
So you'll see that the device file for the primary partition
/dev/hda1-4
or
/dev/sda1-4
not
hda5
sda5
Extended partitions are intended to extend more logical partitions, which under Linux consume 12 numbers such
sda5-16
hda5-16
It specifies the type of file system on this partition. Linux supports many file system types such as msdoc, vfat, ext2, ext3, and more information is covered further in the next section.
The partition is analyzed and understood by analyzing the first 512 bytes of the hard disk, or
MBR
Let's take a look at this picture:
It is used to describe the structure of the
MBR
MBR
includes the boot section, partition table, and end tag'(55AAH), which consumes 446 bytes, 64 bytes, and 2 bytes of 512 bytes, respectively.
Focus here only on the partition table portion, the 64-byte in the middle and the left part of the figure.
Since I'm
SCSI
hard drive, copy the first 512 bytes of the hard drive from my
/dev/sda
device
mbr.bin
file.
$ sudo -s
# dd if=/dev/sda of=mbr.bin bs=512 count=1
The MBR data is analyzed with commands such as
file
od
fdisk
and compared to the figure above to deepen understanding.
MBR
$ file mbr.bin
mbr.bin: x86 boot sector, LInux i386 boot LOader; partition 3: ID=0x82, starthead 254, startsector 19535040, 1959930 sectors; partition 4: ID=0x5, starthead 254, startsector 21494970, 56661255 sectors, code offset 0x48
$ od -x mbr.bin | tail -6 #仅关注中间的64字节,所以截取了结果中后6行
0000660 0000 0000 0000 0000 a666 a666 0000 0180
0000700 0001 fe83 ffff 003f 0000 1481 012a 0000
0000720 0000 0000 0000 0000 0000 0000 0000 fe00
0000740 ffff fe82 ffff 14c0 012a e7fa 001d fe00
0000760 ffff fe05 ffff fcba 0147 9507 0360 aa55
$ sudo -s
# fdisk -l | grep ^/ #仅分析MBR相关的部分,不分析逻辑分区部分
/dev/sda1 * 1 1216 9767488+ 83 Linux
/dev/sda3 1217 1338 979965 82 Linux swap / Solaris
/dev/sda4 1339 4865 28330627+ 5 Extended
file
of the file command show that the newly copied 512 bytes are boot sectors, with the parts separated by
bootloader
partition 3, and partition 4. T
he type of partition 3 is 82,
swap
partition (which can be
l
l command of the
fdisk
command), which corresponds to column 5
/dev/sda3
in
fdisk
result, and the number of sectors in partition 3 is 195993 0, the number of bytes converted to
1959930\*512
(currently, the default sector size of the hard disk is 512 bytes),
swap
partition is 1024 bytes, so the number of blocks is
:
$ echo 1959930*512/1024 | bc
979965
It is exactly the number of blocks corresponding to the fourth column of the row in the
fdisk
result
/dev/sda3
and similarly, partition 4
fdisk
results of fdisk and
file
Let's take a look at the results of the
od
command displayed in hete sixteenths, and consider partition 3, and calculate that the result of
od
partition 3 is:
fe00 ffff fe82 ffff 14c0 012a e7fa 001d
The first is the partition
00H
from the figure above, it is not the boot
80H
is the boot partition), and the partition type?
For
82H
which is consistent
file
display, let's look at the partition size, which is the number of sectors in the
file
result.
$ echo "ibase=10;obase=16;1959930" | bc
1DE7FA
Just corresponds
e7fa 001d
also considers the results of the boot partition:
0180 0001 fe83 ffff 003f 0000 1481 012a
Partition tag:
80H
reflects that the partition is the boot partition, followed by the disk sector in which the boot partition is located, 010100, i.e. 1 side 0 channel 1 sector.
Other content can be analyzed against it.
For time, please refer to the information below for more details or check out the relevant manual for the system.
Supplement: When installing a system, you can
fdisk
such as
cfdisk
and so on.
If you want to start from a partition, you need to mark
80H
for example, by setting a partition to
bootable
cfdisk
Resources:
Before the introduction of logical volumes, partition types and file system types could be treated almost equally, and the process of setting partition types was the process of formatting partitions and establishing the appropriate file system types.
The following mainly describes how to establish a link between partition and file system type, that is, how to format partitioning to the specified file system type.
Let's take a look at the common types of file systems under Linux (if you want to see all the file types supported by Linux, you can view them
l
command of the
fdisk
command, or you can view them via
man fs
or you can view
/proc/filesystems
ext2
ext3
ext4
These three are the types typically used by Linux root file systems
swap
This is a file system used to implement Linux virtual memory, which typically requires a dedicated partition to be established
and
swap
file system (if you want to add
swap
partitions, you can refer to the resources in this section
dd
mkswap
swapon
swapoff
etc.)
proc
This is a relatively special file system that exists as an interface between the kernel and the user, built in
cat
/proc
system can be viewed by cat commands, and even
/proc/sys
the current premise is that the
proc
file system needs to be mounted on:
mount -t proc proc /proc
In addition to the file system types mentioned above, Linux supports a variety of common file system types, including
vfat
iso
xfs
nfs
and under Linux, you are free to view and operate file systems used by other operating systems such as Windows.
So how do you establish an association between disks and these file system types? Format.
The process of formatting is actually the process of reorthorizing partitions, possible
mkfs
and, of course, by commands such as
fdisk
O
nly
mkfs
which can be used to format an existing partition and not to implement partitioning
mkfs
(if you want to partition and format a disk,
fdisk
When formatted, the data on the partition is organized by a particular file system type.
For example,
/dev/sda9
partition as
ext3
$ sudo -s
# mkfs -t ext3 /dev/sda9
If you want to list the file system types for each partition, you
fdisk -l
command.
For more information, please refer to the following information.
Resources:
The last section formatted partitions directly into some file system type, but given the need to extend new storage devices, developers introduced logical volumes between file systems and partitions. For time, it's no longer detailed here, please refer to: Linux Logical Volume Management Details
The file system ends up with a visual structure that can be presented using commands such as ls, find, tree, etc. It's like an upside-down "tree" that can also be mounted on the nodes of a tree.
Here's a brief introduction to the mounting of the file system.
A file system can mount
mount
through a device to a directory called a mount point. I
nterestingly, under Linux, one directory itself can also be mounted to another directory, and a formatted file can
/dev/loop
(such
iso
file).
In addition, in the case of file systems, Linux supports not only local file systems, but also remote file systems such
nfs
Here are a few examples of file system mounts.
Mounting requires Root permissions, for example, mounting system
/dev/sda1
/mnt
$ sudo -s
# mount -t ext3 /dev/sda1 /mnt/
Looking
/dev/sda1
mount, you can see that a device can mount more than once
$ mount | grep sda1
/dev/sda1 on / type ext3 (rw,errors=remount-ro)
/dev/sda1 on /mnt type ext3 (rw)
For a file system that is already mounted, you can re-mount to support different properties
$ mount -n -o remount, rw /
If the kernel already supports the USB connector, when you plug in the usb stick, you can view the corresponding device number with the
dmesg
command and mount it.
Look at the last few lines in the
dmesg
results, find information like
/dev/sdN
and find the device number for the u disk
$ dmesg
Let's assume that the
vfat
format so that it can also be used on Windows in some print shops
# mount -t vfat /dev/sdN /path/to/mountpoint_directory
For some iso files or iso-formatted discs, you can also mount them with the
mount
command.
For iso files:
# mount -t iso9660 /path/to/isofile /path/to/mountpoint_directory
For discs:
# mount -t iso9660 /dev/cdrom /path/to/mountpoint_directory
# mount -t nfs remote_ip:/path/to/share_directory /path/to/local_directory
# mount -t proc proc /proc
proc
file system is organized in memory, but it can be mounted to a directory. I
t is usually
/proc
directory so that some system management and configuration tools can use it. F
or
top
command uses it to analyze memory usage (reads files
lsmod
such as
/proc/meminfo
and
/proc/modules
/proc/stat
netstat
/proc/net/dev
O
f course, you can also write tools. I
n addition, by
/proc/sys
be dynamically adjusted, such as writing the number 1 to the
/proc/sys/net/ipv4/ip_forward
file to enable the kernel to support packet forwarding.
(For more information,
proc
man``proc
$ mount --bind /path/to/needtomount_directory /path/to/mountpoint_directory
This is interesting, for example, to mount a directory to the root of the ftp service without having to copy the content to share the resources in the directory.
All of the above only mentions mounting, so how to uninstall it? U
se
umount
command to keep up with the mounted source address or mount point (device, file, remote directory, etc.).
For example:
$ umount /path/to/mountpoint_directory
Or
$ umount /path/to/mount_source
If you want to manage a large number of or recurring mount services, then each manual mount is a bad thing. T
he
mount
/etc/fstab
can then be used to write the
mount
corresponding to the mount to the column corresponding to
/etc/fstab
file for
mount -a
and unloading
umount -a
/etc/fstab
are file system, mount point, type, and related options.
More information can be
fstab
the help of
man fstab
Resources:
Linux file system has some of the most basic directories, different directories hold different types of files. T
he most basic directories are
/etc
/lib
/dev
/bin
and so on, which hold system profiles, library files, device files, and executables. T
hese directories are generally required to create such a basic file
busybox
tools such as busybox when doing embedded development.
Here is only a very simple file system, and the file system to carry out a variety of routine operations, in order to deepen the understanding of the file system.
Remember
dd
command?
Use it to produce a fixed-size file, this one for
1M(1024\*1024 bytes)
$ dd if=/dev/zero of=minifs bs=1024 count=1024
Looking at the file
minifs
is a file full of .
\\0
any specific data structure
$ file minifs
minifs: data
Description:
/dev/zero
a very special device, and if you read it, you can
\\0
The file is then formatted as a file system of a specified file type. (
Isn't it amazing that files can also be formatted?)
Yes, not only devices can, but files can also be organized in some file system type, but it is important to note that
ext3
require a minimum
64M
targets.
$ mkfs.ext2 minifs
Look at the file type at this point, and
minifs
are organized in the format of the
ext2
file system
$ file minifs
minifs: Linux rev 1.0 ext2 filesystem data
Because the file is organized in the type of file system,
mount
it with the mount command and use it.
Switch to
root
hook it up and associate it with a special device
/dev/loop
-o loop
$ sudo -s
# mount minifs /mnt/ -o loop
Looking at the file system information, you can see only one directory
lost+found
$ ls /mnt/
lost+found
Under the file system to carry out a variety of general operations, including reading, writing, deletion and so on.
(Save a copy of
minifs
the relevant information can be in-depth analysis of the various operations of the file system changes, so as to gain an in-depth understanding of the file system as a way of organizing data implementation principles, etc.)
$ cp minifs minifs.bak
$ cd /mnt
$ touch hello
$ cd -
$ cp minifs minifs-touch.bak
$ od -x minifs.bak > orig.od
$ od -x minifs-touch.bak > touch.od
After you create a file, compare the similarity and differences between the file system at this time and the previous file system
$ diff orig.od touch.od
diff orig.od touch.od
61,63c61,64
< 0060020 000c 0202 2e2e 0000 000b 0000 03e8 020a
< 0060040 6f6c 7473 662b 756f 646e 0000 0000 0000
< 0060060 0000 0000 0000 0000 0000 0000 0000 0000
---
> 0060020 000c 0202 2e2e 0000 000b 0000 0014 020a
> 0060040 6f6c 7473 662b 756f 646e 0000 000c 0000
> 0060060 03d4 0105 6568 6c6c 006f 0000 0000 0000
> 0060100 0000 0000 0000 0000 0000 0000 0000 0000
Through comparison, it is found that the corresponding location of the file system has changed significantly by adding files
$ echo "hello, world" > /mnt/hello
Execute
sync
command to make sure that the data in the cache has been written to disk
buffer cache
in Figure 1 of this section, where the data in the
cache
is written to disk)
$ sync
$ cp minifs minifs-echo.bak
$ od -x minifs-echo.bak > echo.od
After the contents of the file are written, compare the file system with the previous similarity
$ diff touch.od echo.od
View strings in the file system
$ strings minifs
lost+found
hello
hello, world
Delete
hello
file to see changes to the file system
$ rm /mnt/hello
$ cp minifs minifs-rm.bak
$ od -x minifs-rm.bak > rm.od
$ diff echo.od rm.od
By looking at the strings of the file system, it is found that the contents of the file are not overwritten when they are deleted, so in theory the contents are recoverable at this time
$ strings minifs
lost+found
hello
hello, world
The above only demonstrates some of the common tools for analyzing file systems, and analyzes a few general operations, if you want to understand the implementation principles of the file system in great depth, please familiarize yourself with the use of the above tools and read the relevant materials.
Resources:
With
fuse
it is possible to develop a file system in user space, and if you want to develop your own file system, it is recommended to read:
Use fuse to develop your own file system.