copying data from old datastore on to new ESXi

 This is how I copied data from an ESXi datastore on an old m2 ssd to a new ESXi machine:

1. Put the SSD into a USB adaptor
   
   
2. Plugged into new ESXi
   
   
3. Enabled SSH/ESXCLI
   
   
4. Used PuTTY to log into EXSi server
   
   
5. /etc/init.d/usbarbitrator stop
   
   
6. Obtain the UUID of VMFS partition using
esxcli storage vmfs snapshot list


7. then mount it using
esxcli storage vmfs snapshot mount -u <UUID>


8. then log into ESXi GUI and copy files across
   

 

Thanks to:

• http://woshub.com/mount-usb-flash-drive-vmware-esxi/
  
  
• https://serverfault.com/questions/998817/mount-vmfs-partition-in-esxi

ESXi on NUC12 NUC12WSHi5

Happily had ESXi running on a NUC7 for approx 4 years.  The NUC7 recently expired so I thought it would be an easy exercise to replace with a NUC12 and new SSD.... many frustrating hours later...

1. Intel i5-1240P contains e-cores, ESXi doesn't support these, so when booting up ESXi you'll get the "purple screen of death" and warning about CPU mismatches.  Internet advice is to disable the e-cores in the BIOS however there is no option to do this in the NUC12WSHi5 BIOS.  Instead press SHIFT+O at the start of the ESXi startup sequence.  I then added the ESXi kernel option "cpuUniformityHardCheckPanic=FALSE" to the end of the editable line (so I had "runweasel cdromBoot cpuUniformityHardCheckPanic=FALSE") and pressed enter.  ESXi will now boot ok (with a flash of red text).  To be honest I'm pretty uncomfortable about ignoring warnings but it's just a test rig and my only other option is to buy another machine.

2. Assuming you're using ESXi 7 or lower the networking isn't going to work.  This is because the drivers for the newer networking chips aren't in ESX 7 base package.  You can either download/install the community drivers or use ESXi 8+ which includes the drivers.  I did the later for ease.  Good article here: https://williamlam.com/2022/09/esxi-on-intel-nuc-12-pro-wall-street-canyon.html

3. Next I find that the ESXi does not reboot, it complains "No bootable device detected".  Forum suggestions are this is a problem writing to the UEFI NVRAM.  Not my strong area but some suggestions to use (older) MBR rather than GPT.  So I reinstall, press SHIFT+O and this time add "formatwithmbr" (so I now have "runweasel cdromBoot cpuUniformityHardCheckPanic=FALSE formatwithmbr").  It works, but I can't say I fully understand why and wouldn't dream of doing this on a production system.

We'll see how things go.  I'll report back any issues encountered below this line.

EDIT 1: so the NUC12WSHi5 has an internal fan.  It's quiet but not as quiet as my previous fanless NUC7 and that's an annoyance as I keep these machines in a bedroom cupboard.

MySQL Data truncation: Incorrect datetime value

 This got me going today.  Trying to alter a datetime column to timestamp.  Fails on a single row with date '2011-10-02 02:29:42'

Gives the following error message:

SQL State  : 22001
Error Code : 1292
Message    : Data truncation: Incorrect datetime value: '2011-10-02 02:29:42' for column 'date' at row 6151

I check the format of the date again and again.  And again.  It's definitely ok.  Change the year and it works, but change the seconds on it doesn't.  WTF is going on?

Well, it's daylight saving.  Check internet and sure enough in Australia daylight saving change on  2011-10-02 moved the clocks forward one hour from 2am to 3am.  So any timestamp value between 2am and 3am is invalid (in this timezone)! 

FRITZ!Box + WiFi printer

Using a Fritzbox 7490 I couldn't detect a WiFi printer from my laptop.

Printer suggested checking the "privacy separator" setting.

Sure enough, by default, the Fritzbox does not allow WiFi devices to communicate with each other.  This is good security practice, but until I can wire in my printer I need to disable.  It's done like this:

Select "The active wireless devices below may communicate with each other", and press "Apply".  Bingo, I can print again.

Mosquitto 2.0 + mosquitto-auth-plugin + MySQL

The de facto Mosquitto auth plugin jpmens/mosquitto-auth-plug has been archived since March 2019.  Unfortunately it wasn't handed over to anyone and, as of today, 475+ forks of the project exist.  I decided to get one of these working, although on reflection a better choice may have been the iegomez/mosquitto-go-auth plugin which is currently maintained.  Anyway, this is how I've got the original working on Ubuntu 18.04:

You may or may not need to install the following packages:

    # apt install gcc g++ make xsltproc docbook-xsl libwebsockets-dev libmysqlclient-dev

If an older version of Mosquitto has already been installed, fully remove it:

    # apt purge mosquitto

Pull down the official 2.0 source:

    # wget https://mosquitto.org/files/source/mosquitto-2.0.10.tar.gz
    # tar xvf mosquitto-2.0.10.tar.gz

Inside mosquitto-2.0.10 directory edit config.mk setting:

    WITH_WEBSOCKETS:=yes
    WITH_CJSON:=no

Websockets is only required if you are using websocket connections to Mosquitto.  Somewhat frustratingly I had to set WITH_CJSON:=no otherwise I get a "can't find cjson.h'" type error on Ubuntu 18.04.  Didn't have enough time to investigate further.

Build and install:

    # make
    # make install

Create a mosquitto user and chown certain directories:

    # useradd -r mosquitto
    # mkdir /var/log/mosquitto
    # chown mosquitto:mosquitto /var/log/mosquitto/
    # mkdir /var/lib/mosquitto
    # chown mosquitto:mosquitto /var/lib/mosquitto/

Create the file /etc/systemd/system/mosquitto.service:

    [Unit]
    Description=Mosquitto MQTT v3.1/v3.1.1 server
    Wants=network.target
    Documentation=http://mosquitto.org/documentation/

    [Service]
    Type=simple
    User=mosquitto
    Group=mosquitto
    ExecStart=/usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
    Restart=on-failure
    SyslogIdentifier=Mosquitto

    [Install]
    WantedBy=multi-user.target



Pull down this (or your choice of) version of the auth plugin:

    # git clone https://github.com/kmihaylov/mosquitto-auth-plug.git

Inside the mosquitto-auth-plug directory edit a copy of config.mk

    # cp config.mk.in config.mk
    # vi config.mk

Update the following paths as appropriate to your environment

    MOSQUITTO_SRC = <your path>/mosquitto-2.0.10
    OPENSSLDIR = <your path>

hint: "which openssl" should reveal the OpenSSL directory (/usr/bin on my machine).

Next build the auth-plugin:

    # make

You will probably get some errors related to types and include files, for example:

/usr/local/include/mosquitto_plugin.h:167:46: error: unknown type name ‘mosquitto_plugin_id_t’; did you mean ‘mosquitto_property’?

The fix is very simple: rearrange the includes in auth-plug.c and log.c as follows:

    #include <mosquitto_broker.h>
    #include <mosquitto_plugin.h>
    #include <mosquitto.h>

It's probably worth copying the new auth-plug.so to a well known location, for example:

           # cp auth-plug.so /var/lib/mosquitto

Copy /etc/mosquitto/mosquitto.conf.example to /etc/mosquitto/mosquitto.conf, and add the following line at the bottom:

    include_dir /etc/mosquitto/conf.d

In /etc/mosquitto/conf.d, create auth-plug.conf:

    auth_plugin /<your path>/auth-plug.so
    auth_opt_backends mysql
    auth_opt_log_quiet false
    auth_opt_host <your mysql host>
    auth_opt_port <your mysql port>
    auth_opt_dbname <your mysql schema>
    auth_opt_user <your mysql user>
    auth_opt_pass <your mysql password>
    auth_opt_userquery <your mysql select>
    auth_opt_ssl_enabled true

As this file contains a password, make readble to the mosquitto user only:

    # chown mosquitto:mosquitto auth-plug.conf
    # chmod go-rwx auth-plug.conf

This should be enough to get you going, but beware of these gottchas:

1/ you need the auth_opt_backends option set appropriately.  If you simply cut and paste from the jpmens page you'll likely miss this

2/ in the PBKDF2 examples the salt value is confusing, for example:

    PBKDF2$sha256$901$x8mf3JIFTUFU9C23$Mid2xcgTrKBfBdye6W/4hE3GKeksu00+

The actual salt is "x8mf3JIFTUFU9C23", the auth plugin does not base64 decode it, however the password hash *is* base64 decoded.  Bear this in mind when you populate your database. 

3/ Like me, you may have switched from password file authentication.  If so, make sure you remove the "password_file" option from your mosquitto.conf or else it will continue to use the password file and you'll waste time wondering why the auth plug isn't working.

4/ If the auth-plugin is refusing to authenticate your clients edit the config.mk and add -DPWDEBUG to the CFG_CFLAGS option.  Rebuild the plugin and you'll get additional, helpful, logging in /var/log/mosquitto/mosquitto.log

5/ Mosquitto only:  persistence is configured but still doesn't remember messages?  /var/lib/mosquitto/mosquitto.db not getting any larger?  Check your disk space.  It seems to silently fail when your disk is full.

Finally it does work, it can be frustrating at times, but hang in there.  I hope the above helps.  I have had it working with PBKDF2WithHmacSHA1 and PBKDF2WithHmacSHA256 + many different iterations, salts and key lengths.  It's fantastic.

Kiln -> Jenkins web hooks (HTTP ERROR 403 No valid crumb was included in the request)

Our Kiln web hooks stopped working a while ago, finally had a chance to fix.  Here's how:

In Jenkins configure a user and add an API Token (Dashboard->people->user->configure->API Token).  Add a new token.  Name not important so long as it makes sense to you.  Copy the token string - it will not be displayed again:

 

In Kiln go to Web Hooks (top right, click on avator, in top down menu Kiln Administration section) - note you will need admin right to do this.  Create a new web hook:

Note "JENKINS USERNAME" must match the user used in Jenkins above and "PASSWORD" should be the token string generated above (note: not the user's usual password).

Thunderbird vs eM Client: an experiment

A while back my Thunderbird email client started crashing.  It mostly happened after I'd spent a long time composing an email and then bang, gone.  A lot of time wasted re-composing the email.  

Searched for and decided on eM Client as an alternative.  Looked decent and I had no issues paying for a license.  I trialed it for 6 months but eventually switched back to Thunderbird.  Overall I liked eM Client  but just couldn't get past the following:

1/ Slow startup.  Those extra few seconds looking at the orange splash screen began to really annoy.  

2/ Opening individual emails also seems to have unnecessary delay

3/ Custom "From" email addresses.  Very easy to do with Thunderbird from the composition window, however with eM Client an alias first needs to be set up.  Again, at first this doesn't seem too bothersome, but after a while...arghh!

4/ unread emails filter.  click and it selects unread emails, but as soon as you read one of these emails it throws you back to where the email was in your inbox.  This is very frustrating when trying to read all your unread emails as you have to keep clicking unread emails.  Thunderbird's implementation is way superior.

On the flip-side it never crashed and the CardDAV support is superior (note Thunderbird beta finally has native support for this).

Anyway I'm back with Thunderbird.  And it's crashed on me again.  I think it's an issue saving drafts to remote email server, so I've switched to local folders for drafts which will hopefully resolve the issue.