Boot process
After system boot, the 1st process (pid=1) /sbin/init is launched.
/sbin/init is actually a symbolic link to a specific implementations. e.g. sysvinit (old), systemd, upstart ..
systemctl starts/stops/checks status of program lanched by /sbin/init implementation
- grub is the most used boot loader on current distributions.
- check version: grub-probe --version
- Install grub on a device: grub-install
- grub configuration can be changed by editing "/etc/default/grub" file. Then, "update-grub" command should be executed.
'quiet' is used as boot parameter to suppress boot messages (example: can be added as part of grub config).
- check version: grub-probe --version
- Install grub on a device: grub-install
- grub configuration can be changed by editing "/etc/default/grub" file. Then, "update-grub" command should be executed.
'quiet' is used as boot parameter to suppress boot messages (example: can be added as part of grub config).
Kernel components:
udev
- Device manager (connect/remove a device; driver / firmware)
- Check /etc/udev
- Mounted to /dev (temporary file system) on startup
- Process intercommunication framework
- Provide OOP api
- Virtual file system
- Mounted to /sys
- Presents information about the various kernel subsystems such as hardware devices and drivers.
- Virtual file system
- Mounted to /pro
- Provide informations about processes
lsmod: lists modules currently loaded (/proc/modules)
lspci: lists PCI devices currently in use
Remarks:
Besides of device driver, some hardware also requires firmware or microcode to be loaded into the device before it can become operational.
A PCI device is any piece of computer hardware that plugs directly into a PCI slot on a computer's motherboard.
Basically, two primary buses are linked to the cpu:
- The system bus: fastest, connects RAM, to the CPU.
- The PCI bus: facilitate communication of hardware devices with the microprocessor.
Run levels
A reference to a set of scripts that have to be executed when that level is invoked:
- run levels (old sysvinit): the reference is a number (0 to 6). Scripts are defined under /etc (rc0.d, rc1.d ... directories)e.g. "net 0" to shutdown, "telinit 1" to reboot
- Systemd run levels (new approach): the reference is a "target name". Scripts are defined under /lib/systemd/system/runlevel*
Remark:
0 run level takes effect when shutdown
1 to 5 run levels take effect when boot
6 run level takes effect when reboot
File types
- - : regular file
- d : directory
- c : character device file
- b : block device file
- s : local socket file
- p : named pipe
- l : symbolic link
Partitions
Linking libraries
Linux uses dynamic linking to hook dependencies of a program.
Check "/etc/ld.so.conf.d"
- Use "ldd" to check program libraries, e.g "ldd /bin/ls"
- Use "ldconfig" to add your libraries to the Linux Linker.
- You can temporarily add directories to your shared library path with export LD_LIBRARY_PATH.
- /etc/default/grub is the file used to set grub default options
- Shared libraries are listed in /etc/ld.so.conf
Package management
- dpkg: debian package manager
- apt: advance packaging tool (on top of dpkg)
- rpm: redhat package manager
- yum: yellowdog updater modified (on top of rpm). /etc/yum.repos.d
- "env" : lists environment variables
- "man man" : each on the manual itself
- 'expand" / ("unexpand") : given a file, convert from tabs to space / (from space to tabs)
- "wc" : prints umre of lines, words and bytes in the file
- "fmt" : splits lines of the files to be readable.
- "head" / "tail" : list top / bottom lines of the file
- "less" file pagination
- "sed" : stream editor for filtering and transforming text
- "touch" to change timestamp of file
- "dd" data duplicate, copy/dump a file (or a portion of a file)
- "tee" ouputs to stdin and given file
STDIN: file descriptor 0
STDOUT: file descriptor1
STDERR: file descriptor2
while piping, xargs maps/applies (list.map) a command to each element.
Jobs (non shell command)
- "jobs", "bg", "fg" are terminal commands, not shell commands.
- a job is any ruining or stopped (ctrl+z) TERMINAL execution.
- "jobs" : lists ruining / stopped terminal executions
- contrast : job is related to the Terminal , whereas process (ps) is any executed program at OS level. A process is wrapped in a job to be controlled (stop, background, foreground ...) by terminal. A job is not a process, it's a terminal proxy to send signals to rapped process
- "nohup" command turn off background noises
Processes
"ps" displays process run y the current user
"ps ax" : a = all processes wherever (e.g all terminals) they are executed (= a), for all users (= x)
"ps axu"! same as above, with friendly display (= u)
''kill" send signals to a process:
Interrupt a process (graceful mode) --> kill -INT <pid>
Kill a process (force mode) --> kill -KILL <pid>
Stop a process (i.e ctrl z) --> kill -STOP <pid>
Terminate a process (graceful + could ignore signal) --> kill -TERM <pid>
(when you change state of a process, the same state is reflected on its associated job, if any)
kill by name: "killall" (exact matching) safe than "pkill"
Define / Alter the priority of a process (cpu scheduling :
"nice" / 'renice"
'-20' is the highest niceness you can assign with the nice command.
19 is the lowest niceness you can assign with the nice command.
"fuser <directory>" shows processes using directory
"fuser <directory>" shows processes using directory
Monitoring
- "top" : dynamic real-time view of running system. displays system summary information + list of processes or threads currently being managed by the Linux kernel.
"free" : information about total, free and used memory
"uptime" : current time + how long the system has been running + how many users are currently logged + the system load averages for the past 1, 5, and 15 minutes.
System load averages: the average number of processes that are either in a runnable or uninterruptable state. The averages are taken over the three time intervals.
example: a load average of 1 means a single CPU system is loaded all the time while on a 4 CPU system it means it was idle 75% of the time.
Text processing
contrast:
grep : requires to backslash quantifiers (+, ?, *)
egrep; doesn't requires to backslash quantifier
fgrep: doesn't support quantifier
File editing
VI / VIM:
- search: /, ?
- cut line: dd
- copy line; yy
- paste line: p or P
- remove character; x
- shift +ZZ shortcut of ":wq"
File systems
Filesystem refers to the device portion holding the data. The device portion can be logical (partition or lvm of a hard disk) or physical (usb, cd rom ..)
Mountpoint is a logical concept of mapping of a filesystem device to a path.
"df " : displays available / used space for all mounted filesystem (device portions).
"lsblk": lists information about all (mounted or not) device portions.
"du" : Estimates disk usage of a file or directory.
In practice: "du -h --max-depth=1 <dir>
"fdisk /dev/sda" : create, change & delete partitions in "sda" device.
"parted" alternative to "fdisk"
Partitions info are managed and stored on partition table (into the device).
There are two data models for partition table: MBR (old) & GPT.
"mkfs /dev/sda ext2" : formatting a partition (aka: put a filesystem on it),
Types:
"mkswap" create a swap space on the given partition
"swapen" add given partition to the swap system space
File permissions
"ln" : creates hard link to an inode
"ln -s" creates a soft link to an inode (soft links can be created cross filesystems)
Track file systems and binaries location
"which" tell you the path to a program
"type" let you if a command is a shell built-in
"whereis"
"locate" ("updatedb")
Remarks:
- "find . -inum <inode number>" will search the current directory for a particular inode number.
- "edquota" is used to edit a users quota.
- "chmod a+x <file>" : The 'a' in 'a+x' will apply to the user, group, and other. It stands for 'all'.
- Every user has a default or primary group. When a user logs in, the group membership is set for their primary group. This means that when a user launches a program or creates a file, both the file and the running program will be associated with the user’s current group membership.
- To run programs or create a file in a different group, the user must run the "newgrp" command to switch their (login scope) current group.
- By default, all files are “owned” by the user who creates them and by that user’s default group.
- Every user has a default or primary group. When a user logs in, the group membership is set for their primary group. This means that when a user launches a program or creates a file, both the file and the running program will be associated with the user’s current group membership.
- To run programs or create a file in a different group, the user must run the "newgrp" command to switch their (login scope) current group.
- By default, all files are “owned” by the user who creates them and by that user’s default group.
Shell config
- "#!" hash bang
- "set -o xtrace" enable verbose mode when running commands.
- "set -o noclobber" will stop the > redirect from overwriting files.
- echo $? will show you the return value of the last command run in the shell.
X11 protocol
https://wiki.ubuntu.com/X/Architecture
- "xwininfo" will show you detailed information about your window when you click on it.
- "/etc/X11/xorg.conf" is traditionally the X Server configuration file name.
- The DISPLAY variable is used to select your X server.
- (run levels) ; "graphical.target" is used to load your graphical interface upon boot.
- startx will start an X windows session.
- xdm is the default barebones display manager (other managers: gdm, kdm ..)
- "xhost+" is used to disable access control and allow any client to connect.
Manage users / groups
"/etc/passwd" : contains users
template : user:pass(always *):userId:primaryGroupId:info:home:defayltShell
"/etc/shadow" : contains (encrypted ) passwords
template : user:password:remaining fields are about creation dates and expiry polity
"/etc/groups" : contains groups
template : nameOfGroup:passwordOfGroup(rarely used):groupId:list of groups
Job scheduling
"at" : executes commands at a specified time.
"atq" : lists the user's pending jobs, unless the user is the superuser; in that case, everybody's jobs are listed. The format of the output lines (one for each job) is: Job number, date, hour, queue, and username.
"atrm" : deletes jobs, identified by their job number.
"batch" : executes commands when system load levels permit; in other words, when the load average drops below 1.5, or the value specified in the invocation of atd.
"crontab" schedule a job to be executed periodically on a specific time
"anacron" same as "crontab", but if the computer is off, then the job is executed once (after a delay) the system is on.
Date & Time
"ntpdate" sets the local date and time by polling the Network Time Protocol (NTP) server(s) given as the server arguments to determine the correct time.
"hwclock" : is an administration tool for the system and hardware clocks .
Syslogs vs Journalization
Syslog is the standard solution for logging on UNIX. (implementations: rsyslog or syslog-ng).
Issues:
- lack of structure: Finding information in large plain text files with lots of unrelated information can be difficult
- Simple plain text files also require log rotation to prevent them from becoming too large. In log rotation, existing log files are renamed and compressed.
Journald replaces the plain text files of syslog with a binary format that:
- Allows for log messages with multiple fields and multi-line text
- Stores these messages in a space-efficient way that does not require renaming files for maintenance
- Gives fast access to messages given specific criteria, much like a database would
Mail transfer agent
Networking
"ifconfig" : displays network interfaces ("ip" command is a new alternative)
"ifdown" "ifdown" activate/deactivate an the given interface
"route" : show / manipulate ip routing table
"host" & "dig" : DNS lookup tools
"netstat" : print network connections
"traceroute" : displays packet route
"netcat" ; It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning,
Common uses include:
· simple TCP proxies
· shell-script based HTTP clients and servers
· network daemon testing
· a SOCKS or HTTP ProxyCommand for ssh
"/etc/passwd" : contains users
template : user:pass(always *):userId:primaryGroupId:info:home:defayltShell
"/etc/shadow" : contains (encrypted ) passwords
template : user:password:remaining fields are about creation dates and expiry polity
"chage" is used to change user password expiry information.
"/etc/groups" : contains groups
template : nameOfGroup:passwordOfGroup(rarely used):groupId:list of groups
"/etc/skel" is copied to a new users home directory when created with the adduser command.
The "getent" command can query database files, eg 'getent services'.
Job scheduling
"at" : executes commands at a specified time.
"atq" : lists the user's pending jobs, unless the user is the superuser; in that case, everybody's jobs are listed. The format of the output lines (one for each job) is: Job number, date, hour, queue, and username.
"atrm" : deletes jobs, identified by their job number.
"batch" : executes commands when system load levels permit; in other words, when the load average drops below 1.5, or the value specified in the invocation of atd.
"crontab" schedule a job to be executed periodically on a specific time
"anacron" same as "crontab", but if the computer is off, then the job is executed once (after a delay) the system is on.
Date & Time
"ntpdate" sets the local date and time by polling the Network Time Protocol (NTP) server(s) given as the server arguments to determine the correct time.
"hwclock" : is an administration tool for the system and hardware clocks .
"ntpd" will run in the background and keep your system clock accurate.
"date +%s" will show the date in Unix time.
Syslogs vs Journalization
Syslog is the standard solution for logging on UNIX. (implementations: rsyslog or syslog-ng).
Issues:
- lack of structure: Finding information in large plain text files with lots of unrelated information can be difficult
- Simple plain text files also require log rotation to prevent them from becoming too large. In log rotation, existing log files are renamed and compressed.
Journald replaces the plain text files of syslog with a binary format that:
- Allows for log messages with multiple fields and multi-line text
- Stores these messages in a space-efficient way that does not require renaming files for maintenance
- Gives fast access to messages given specific criteria, much like a database would
"journalctl" will show you all systemd journal logs.
"logrotate" will rotate your logs and will delete logs if they're old enough.
"missingok"
(on conf file) will make logrotate simply move on to the next file if
the file is missing and it will not report an error.
Mail transfer agent
The "newaliases" command will rebuilt the email alias database.
"ifconfig" : displays network interfaces ("ip" command is a new alternative)
"ifdown" "ifdown" activate/deactivate an the given interface
"route" : show / manipulate ip routing table
"host" & "dig" : DNS lookup tools
"netstat" : print network connections
"traceroute" : displays packet route
"netcat" ; It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning,
Common uses include:
· simple TCP proxies
· shell-script based HTTP clients and servers
· network daemon testing
· a SOCKS or HTTP ProxyCommand for ssh
- The nameservers to be used by the system are listed in /etc/resolv.conf.
- ping uses ICMP packets.
- ping will test the latency between two computers.
- /etc/hosts is where you can list your own alias entries for IP addresses and hosts.
- traceroute and tracert do the same thing, but traceroute is the Linux command. tracert is the Windows command.
- route -n will display the route table without resolving IP addresses.
- nsswitch will list in order which services to use to resolve hostnames.
SSH
"ssh-eygen" : to create a keypair (private + public keys)
"ssh-copy-id" : use locally available keys (public) to authorise logins on a remote machine
Others
"nmap <host>" : scans open ports on the given host.
"lsof" show open files (regular file, directory, socket ..) and which process keep them open (similar to "fuser")
- "id" print user and group information for the specified USER, or (when USER omitted) for the current user.
- "ulimit -u" will set the maximum number of user processes.
"ssh-eygen" : to create a keypair (private + public keys)
"ssh-copy-id" : use locally available keys (public) to authorise logins on a remote machine
Others
"nmap <host>" : scans open ports on the given host.
"lsof" show open files (regular file, directory, socket ..) and which process keep them open (similar to "fuser")
- "id" print user and group information for the specified USER, or (when USER omitted) for the current user.
- "ulimit -u" will set the maximum number of user processes.
- /etc/nologin is the correct file. See man 5 nologin for further information.
- While tcpd will launch the appropriate program after logging it, xinetd
is the application doing the monitoring and launching of tcpd and other
applications.
- Most of the rc scripts that are used by Systemd to start and stop daemons are stored in the "/usr/lib/systemd/system" directory and called [daemon].service.
- When rsyslog daemon is loaded upon system startup, it creates a socket, /dev/log, for other system processes to write to. It then reads any information written to this socket and saves the information in the appropriate log file according to entries in the /etc/rsyslog.conf file and any files within the /etc/rsyslog.d directory.
- The Linux kernel emits several messages before the system is able to run syslogd and/or possibly klogd to grab them. Since the messages may still be of value, the Linux kernel stores them internally. The way you can access them is by using the dmesg command.
- The other programs (other than "man -k") that search the man database are apropos or whatis.
- Most of the rc scripts that are used by Systemd to start and stop daemons are stored in the "/usr/lib/systemd/system" directory and called [daemon].service.
- When rsyslog daemon is loaded upon system startup, it creates a socket, /dev/log, for other system processes to write to. It then reads any information written to this socket and saves the information in the appropriate log file according to entries in the /etc/rsyslog.conf file and any files within the /etc/rsyslog.d directory.
- The Linux kernel emits several messages before the system is able to run syslogd and/or possibly klogd to grab them. Since the messages may still be of value, the Linux kernel stores them internally. The way you can access them is by using the dmesg command.
- The other programs (other than "man -k") that search the man database are apropos or whatis.
No comments:
Post a Comment