Introduction to ESP8266/EX development

In this post I'll present a step-by-step beginners guide for setting up a software development environment for ESP8266/EX.
Hardware issues are out of the scope for this post, and my assumption is that you have a working unit and a hardware utility you're familiar flashing it with.

The purpose of this post is to help you to skip the error mess resulted when trying to compile an open source project.

This guide will focus on a Linux setup, as it's pretty trivial for this kind of work.

The following components will be required:

• Cross compiler - as ESP runs on a Xtensa architecture
• SDK
• Firmware sources
• Flashing utilities
• ESP unit

Begin with installing the dependencies packages:

sudo apt-get update
sudo apt-get install make unrar autoconf automake libtool gcc g++ gperf flex bison texinfo gawk ncurses-dev libexpat-dev python python-serial sed git

Afterwards, clone the git esp-open-sdk:

cd /home/username/Documents
mkdir Espressif
cd Espressif
git clone --recursive https://github.com/pfalcon/esp-open-sdk.git

Building it might take a few minutes, and is done through:

cd esp-open-sdk
make STANDALONE=y

Note: when newer SDKs are released, updating the SDK is as simple as:

make clean
sudo git pull
sudo git submodule sync
sudo git submodule update

At this stage you have the requirements for building firmwares. This sample will show how to build the esphttpd project.
clone the httpd project:

cd ../
git clone --recursive https://github.com/izhak2/esphttpd
cd esphttpd

Edit the makefile according to the environment you just set up:

XTENSA_TOOLS_ROOT ?= /home/username/Documents/Espressif/esp-open-sdk/xtensa-lx106-elf/bin
SDK_EXTRA_INCLUDES ?= /home/username/Documents/Espressif/esp-open-sdk/sdk/include
SDK_EXTRA_LIBS ?= /home/username/Documents/Espressif/esp-open-sdk/sdk/lib
SDK_ROOT ?= /home/username/Documents/Espressif/esp-open-sdk/

Since during build xtensa-lx106-elf-nm is expected to be called as xt-nm, copy it:

cp /home/username/Documents/Espressif/esp-open-sdk/xtensa-lx106-elf/bin/xtensa-lx106-elf-nm /home/username/Documents/Espressif/esp-open-sdk/xtensa-lx106-elf/bin/xt-nm

Make the binaries available through the PATH environment variable:

PATH=/home/username/Documents/Espressif/esp-open-sdk/xtensa-lx106-elf/bin:$PATH

Finally, begin building. Building has two parts - there's the firmware and the mkespfsimage.
Both depend on the Heatshrink project, but with different settings.
Once completed building mkespfsimage, clean the Heathsrink build prior to building the firmware. Otherwise you might experience linking errors with some memory allocation functions.

First build mkespfsimage. It's used to generate the espfs (file-system over a single file, which is mapped as a whole to the flash).

cd mkespfsimage
make

and as explained, clean:

cd ../
make clean

Change heatshrink configuration to fit the firmware build; edit:

/home/username/Documents/esphttpd/lib/heatshrink/heatshrink_config.h

and perform the following changes:
"malloc" to "os_malloc"
"free" to "os_free"

Now, build the firmware using:

make rawflash ota=true

The make rawflash step will result in the firmware and espfs files available at /firmware, and a try to flash the files through serial.

You may also flash the files manually. For that procedure you may use (windows for now):

/utils/flash/ESP8266Flasher.exe

Additional files, except for firmware and espfs ,required for flashing will be found at:

/home/username/Documents/Espressif/esp-open-sdk/esp_iot_sdk_v1.3.0/bin
blank.bin
boot_v1.4(b1).bin

Map the offsets in ESP8266Flasher as following and flash to the appropriate COM port.
boot_v1.4(b1).bin at 0x00000
user1.new.bin at 0x01000
website.espfs at 0x41000
blank.bin at 0x7E000

Congratulations, now you have the firmware running on the ESP!

To view debug strings during runtime, if configured to be included, you may use PuTTY with 115200 speed serial configuration, and new line settings:

Terminal -> Implicit CR in every LF

to prevent new lines mess.

Finally, OTA flashing can be performed using the ota_flasher.py utility at /utils/flash/ota_flasher.py.
To perform OTA flashing you should have access to the chip of course, either if it's configured as a hotspot, or if connected to some other network component.
Have the ota_flasher.py and curl.exe as well as the firmware files and espfs at the same directory (verify the file names to match inside the script), and flash with the chip ip address.
On default, for example:

ota_flasher.py 192.168.4.1

When flashing firmware, wait between firmware and file-system flashing, since the chip performs a reset once firmware completed uploading.
Therefore, you may abort the script once asked for 'y' to flash the file-system, then re-execute the script skipping the firmware part, just to check if it recognizes the new boot partition.
Boot mode check at the prior to the flashing is done also for that - to make sure the chip is alive and responding, therefore ready for flashing.

Few more notes on OTA flashing -
Once flashed a firmware, you have to flash a file-system as well; although, you may flash only a file-system without replacing the firmware.

Credits:
Special thanks to my friend Boris.
Thanks to pfalcon who maintains the esp-open-sdk project which provides the relevant scripts for building a complete SDK.