TIL Linux emergency sync

Discussion in 'GNU/Linux' started by melbo, Aug 16, 2015.

  1. melbo

    melbo Hunter Gatherer Administrator Founding Member

    I didn't realize that in the event of a system freeze, you can still talk to the kernel and give it some final instructions prior to a hard power cycle of the machine. This allows you to attempt to sync all mounted filesystems before a shutdown and possibly avoid file corruption. Thanks to @sec_monkey for the tip!

    How do I use the magic SysRq keys in emergency?
    You need to use following key combination in order to reboot/halt/sync file system etc:

    The 'SysRq' key is also known as the 'Print Screen' key. COMMAND-KEY can be any one of the following (all keys need to hit simultaneously) :
    • 'b' : Will immediately reboot the system without syncing or unmounting your disks.
    • 'o' : Will shutdown your system off (if configured and supported).
    • 's': Will attempt to sync all mounted filesystems.
    • 'u' : Will attempt to remount all mounted filesystems read-only.
    • 'e' : Send a SIGTERM to all processes, except for init.
    • 'h': Show help, indeed this the one you need to remember.
    So whey you need to tell your Linux computer to reboot or when your X server is crashed or you don't see anything going across the screen then just press:

    ALT+SysRQ+s : (Press and hold down ALT, then SysRQ (Print Screen) key and press 's') -Will try to sync all mounted system

    ALT+SysRQ+r : (Press and hold down ALT, then SysRQ (Print Screen) key and press 'r') -Will reboot the system.

    If you wish to shutdown the system instead of reboot then press following key combination:

    More commands from the man page:
    1 Linux Magic System Request Key Hacks
    2 Documentation for sysrq.c
    4 * What is the magic SysRq key?
    5 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    6 It is a 'magical' key combo you can hit which the kernel will respond to
    7 regardless of whatever else it is doing, unless it is completely locked up.
    9 * How do I enable the magic SysRq key?
    10 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    11 You need to say "yes" to 'Magic SysRq key (CONFIG_MAGIC_SYSRQ)' when
    12 configuring the kernel. When running a kernel with SysRq compiled in,
    13 /proc/sys/kernel/sysrq controls the functions allowed to be invoked via
    14 the SysRq key. The default value in this file is set by the
    15 CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE config symbol, which itself defaults
    16 to 1. Here is the list of possible values in /proc/sys/kernel/sysrq:
    17 0 - disable sysrq completely
    18 1 - enable all functions of sysrq
    19 >1 - bitmask of allowed sysrq functions (see below for detailed function
    20 description):
    21 2 = 0x2 - enable control of console logging level
    22 4 = 0x4 - enable control of keyboard (SAK, unraw)
    23 8 = 0x8 - enable debugging dumps of processes etc.
    24 16 = 0x10 - enable sync command
    25 32 = 0x20 - enable remount read-only
    26 64 = 0x40 - enable signalling of processes (term, kill, oom-kill)
    27 128 = 0x80 - allow reboot/poweroff
    28 256 = 0x100 - allow nicing of all RT tasks
    30 You can set the value in the file by the following command:
    31 echo "number" >/proc/sys/kernel/sysrq
    33 The number may be written here either as decimal or as hexadecimal
    34 with the 0x prefix. CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE must always be
    35 written in hexadecimal.
    37 Note that the value of /proc/sys/kernel/sysrq influences only the invocation
    38 via a keyboard. Invocation of any operation via /proc/sysrq-trigger is always
    39 allowed (by a user with admin privileges).
    41 * How do I use the magic SysRq key?
    42 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    43 On x86 - You press the key combo 'ALT-SysRq-<command key>'. Note - Some
    44 keyboards may not have a key labeled 'SysRq'. The 'SysRq' key is
    45 also known as the 'Print Screen' key. Also some keyboards cannot
    46 handle so many keys being pressed at the same time, so you might
    47 have better luck with "press Alt", "press SysRq", "release SysRq",
    48 "press <command key>", release everything.
    50 On SPARC - You press 'ALT-STOP-<command key>', I believe.
    52 On the serial console (PC style standard serial ports only) -
    53 You send a BREAK, then within 5 seconds a command key. Sending
    54 BREAK twice is interpreted as a normal BREAK.
    56 On PowerPC - Press 'ALT - Print Screen (or F13) - <command key>,
    57 Print Screen (or F13) - <command key> may suffice.
    59 On other - If you know of the key combos for other architectures, please
    60 let me know so I can add them to this section.
    62 On all - write a character to /proc/sysrq-trigger. e.g.:
    64 echo t > /proc/sysrq-trigger
    66 * What are the 'command' keys?
    67 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    68 'b' - Will immediately reboot the system without syncing or unmounting
    69 your disks.
    71 'c' - Will perform a system crash by a NULL pointer dereference.
    72 A crashdump will be taken if configured.
    74 'd' - Shows all locks that are held.
    76 'e' - Send a SIGTERM to all processes, except for init.
    78 'f' - Will call oom_kill to kill a memory hog process.
    80 'g' - Used by kgdb (kernel debugger)
    82 'h' - Will display help (actually any other key than those listed
    83 here will display help. but 'h' is easy to remember :)
    85 'i' - Send a SIGKILL to all processes, except for init.
    87 'j' - Forcibly "Just thaw it" - filesystems frozen by the FIFREEZE ioctl.
    89 'k' - Secure Access Key (SAK) Kills all programs on the current virtual
    90 console. NOTE: See important comments below in SAK section.
    92 'l' - Shows a stack backtrace for all active CPUs.
    94 'm' - Will dump current memory info to your console.
    96 'n' - Used to make RT tasks nice-able
    98 'o' - Will shut your system off (if configured and supported).
    100 'p' - Will dump the current registers and flags to your console.
    102 'q' - Will dump per CPU lists of all armed hrtimers (but NOT regular
    103 timer_list timers) and detailed information about all
    104 clockevent devices.
    106 'r' - Turns off keyboard raw mode and sets it to XLATE.
    108 's' - Will attempt to sync all mounted filesystems.
    110 't' - Will dump a list of current tasks and their information to your
    111 console.
    113 'u' - Will attempt to remount all mounted filesystems read-only.
    115 'v' - Forcefully restores framebuffer console
    116 'v' - Causes ETM buffer dump [ARM-specific]
    118 'w' - Dumps tasks that are in uninterruptable (blocked) state.
    120 'x' - Used by xmon interface on ppc/powerpc platforms.
    121 Show global PMU Registers on sparc64.
    123 'y' - Show global CPU Registers [SPARC-64 specific]
    125 'z' - Dump the ftrace buffer
    127 '0'-'9' - Sets the console log level, controlling which kernel messages
    128 will be printed to your console. ('0', for example would make
    129 it so that only emergency messages like PANICs or OOPSes would
    130 make it to your console.)
    132 * Okay, so what can I use them for?
    133 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    134 Well, unraw(r) is very handy when your X server or a svgalib program crashes.
    136 sak(k) (Secure Access Key) is useful when you want to be sure there is no
    137 trojan program running at console which could grab your password
    138 when you would try to login. It will kill all programs on given console,
    139 thus letting you make sure that the login prompt you see is actually
    140 the one from init, not some trojan program.
    141 IMPORTANT: In its true form it is not a true SAK like the one in a :IMPORTANT
    142 IMPORTANT: c2 compliant system, and it should not be mistaken as :IMPORTANT
    144 It seems others find it useful as (System Attention Key) which is
    145 useful when you want to exit a program that will not let you switch consoles.
    146 (For example, X or a svgalib program.)
    148 reboot(b) is good when you're unable to shut down. But you should also
    149 sync(s) and umount(u) first.
    151 crash(c) can be used to manually trigger a crashdump when the system is hung.
    152 Note that this just triggers a crash if there is no dump mechanism available.
    154 sync(s) is great when your system is locked up, it allows you to sync your
    155 disks and will certainly lessen the chance of data loss and fscking. Note
    156 that the sync hasn't taken place until you see the "OK" and "Done" appear
    157 on the screen. (If the kernel is really in strife, you may not ever get the
    158 OK or Done message...)
    160 umount(u) is basically useful in the same ways as sync(s). I generally sync(s),
    161 umount(u), then reboot(b) when my system locks. It's saved me many a fsck.
    162 Again, the unmount (remount read-only) hasn't taken place until you see the
    163 "OK" and "Done" message appear on the screen.
    165 The loglevels '0'-'9' are useful when your console is being flooded with
    166 kernel messages you do not want to see. Selecting '0' will prevent all but
    167 the most urgent kernel messages from reaching your console. (They will
    168 still be logged if syslogd/klogd are alive, though.)
    170 term(e) and kill(i) are useful if you have some sort of runaway process you
    171 are unable to kill any other way, especially if it's spawning other
    172 processes.
    174 "just thaw it(j)" is useful if your system becomes unresponsive due to a frozen
    175 (probably root) filesystem via the FIFREEZE ioctl.
    177 * Sometimes SysRq seems to get 'stuck' after using it, what can I do?
    178 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    179 That happens to me, also. I've found that tapping shift, alt, and control
    180 on both sides of the keyboard, and hitting an invalid sysrq sequence again
    181 will fix the problem. (i.e., something like alt-sysrq-z). Switching to another
    182 virtual console (ALT+Fn) and then back again should also help.
    184 * I hit SysRq, but nothing seems to happen, what's wrong?
    185 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    186 There are some keyboards that produce a different keycode for SysRq than the
    187 pre-defined value of 99 (see KEY_SYSRQ in include/linux/input.h), or which
    188 don't have a SysRq key at all. In these cases, run 'showkey -s' to find an
    189 appropriate scancode sequence, and use 'setkeycodes <sequence> 99' to map
    190 this sequence to the usual SysRq code (e.g., 'setkeycodes e05b 99'). It's
    191 probably best to put this command in a boot script. Oh, and by the way, you
    192 exit 'showkey' by not typing anything for ten seconds.
    194 * I want to add SysRQ key events to a module, how does it work?
    195 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    196 In order to register a basic function with the table, you must first include
    197 the header 'include/linux/sysrq.h', this will define everything else you need.
    198 Next, you must create a sysrq_key_op struct, and populate it with A) the key
    199 handler function you will use, B) a help_msg string, that will print when SysRQ
    200 prints help, and C) an action_msg string, that will print right before your
    201 handler is called. Your handler must conform to the prototype in 'sysrq.h'.
    203 After the sysrq_key_op is created, you can call the kernel function
    204 register_sysrq_key(int key, struct sysrq_key_op *op_p); this will
    205 register the operation pointed to by 'op_p' at table key 'key',
    206 if that slot in the table is blank. At module unload time, you must call
    207 the function unregister_sysrq_key(int key, struct sysrq_key_op *op_p), which
    208 will remove the key op pointed to by 'op_p' from the key 'key', if and only if
    209 it is currently registered in that slot. This is in case the slot has been
    210 overwritten since you registered it.
    212 The Magic SysRQ system works by registering key operations against a key op
    213 lookup table, which is defined in 'drivers/char/sysrq.c'. This key table has
    214 a number of operations registered into it at compile time, but is mutable,
    215 and 2 functions are exported for interface to it:
    216 register_sysrq_key and unregister_sysrq_key.
    217 Of course, never ever leave an invalid pointer in the table. I.e., when
    218 your module that called register_sysrq_key() exits, it must call
    219 unregister_sysrq_key() to clean up the sysrq key table entry that it used.
    220 Null pointers in the table are always safe. :)
    222 If for some reason you feel the need to call the handle_sysrq function from
    223 within a function called by handle_sysrq, you must be aware that you are in
    224 a lock (you are also in an interrupt handler, which means don't sleep!), so
    225 you must call __handle_sysrq_nolock instead.
    227 * When I hit a SysRq key combination only the header appears on the console?
    228 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    229 Sysrq output is subject to the same console loglevel control as all
    230 other console output. This means that if the kernel was booted 'quiet'
    231 as is common on distro kernels the output may not appear on the actual
    232 console, even though it will appear in the dmesg buffer, and be accessible
    233 via the dmesg command and to the consumers of /proc/kmsg. As a specific
    234 exception the header line from the sysrq command is passed to all console
    235 consumers as if the current loglevel was maximum. If only the header
    236 is emitted it is almost certain that the kernel loglevel is too low.
    237 Should you require the output on the console channel then you will need
    238 to temporarily up the console loglevel using alt-sysrq-8 or:
    240 echo 8 > /proc/sysrq-trigger
    242 Remember to return the loglevel to normal after triggering the sysrq
    243 command you are interested in.
    245 * I have more questions, who can I ask?
    246 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    247 Just ask them on the linux-kernel mailing list:
    248 linux-kernel@vger.kernel.org
    250 * Credits
    251 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    252 Written by Mydraal <vulpyne@vulpyne.net>
    253 Updated by Adam Sulmicki <adam@cfar.umd.edu>
    254 Updated by Jeremy M. Dolan <jmd@turbogeek.org> 2001/01/28 10:15:59
    255 Added to by Crutcher Dunnavant <crutcher+kernel@datastacks.com>
    Howto Reboot or halt Linux system in emergency
    Last edited: Aug 17, 2015
    techsar, sec_monkey, Tobit and 5 others like this.
  2. DarkLight

    DarkLight Live Long and Prosper - On Hiatus Site Supporter

    It looks like the forum software really doesn't like vi/more/less/man output.
    Ganado and sec_monkey like this.
  3. melbo

    melbo Hunter Gatherer Administrator Founding Member

    Ganado and sec_monkey like this.
  4. kckndrgn

    kckndrgn Monkey+++ Moderator Emeritus Founding Member

    Lots of good info, thanks!
    Ganado and sec_monkey like this.
survivalmonkey SSL seal        survivalmonkey.com warrant canary