Solution to problem of module getting marked as [permanent]

Lot of times you will see your own module has been marked as permanent. 

$ lsmod
 Module Size Used by
 hello 78567 0 [permanent] 

      These module can't be removed unless system is rebooted. You get messages like following when you try to remove/rmmod them.

ERROR: Module hello is in use by [permanent]
or
ERROR: Removing 'hello': Device or resource busy

      Solution is quite simple for this problem. Recompile you hello.ko module with -DCC_HAVE_ASM_GOTO flag. Problem is struct module layout has dependency on HAVE_JUMP_LABEL => CC_HAVE_ASM_GOTO => gcc-goto.sh script => gcc version being used. 
      When there is a mismatch, the module exit callback (destructor) gets value of NULL which results in module being marked as permanent. This causes mod->exit in following code snippet of kernel/module.c to become null and because CONFIG_MODULE_FORCE_UNLOAD is not set in config file -EBUSY is returned. 

/* If it has an init func, it must have an exit func to unload */
if (mod->init && !mod->exit) {
     forced = try_force_unload(flags);
     if (!forced) {
         /* This module can't be removed */
         ret = -EBUSY;
         goto out;
     }
 } 

To find out exactly which Linux platforms needs this flag run 

sh scripts/gcc-goto.sh gcc 

command in kernel header directory. If you get output as "y" then you must include this flag in any module compilation