- NETMAP VERSION -
------------------

-*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*-*-*-*-
-A- Tested with linux-3.13.0-121-generic & netmap commit: 8f3f79e4ca168c6d3220e7a5198a5778b030445e,
and,
-B- Tested with linux-3.13.0-121-generic & netmap commit: 21227d47ae995812412d1de22855c7fa8260f53a,
and,
-C- Tested with 4.11.12-200.fc25.x86_64+debug & netmap commit: 21227d47ae995812412d1de22855c7fa8260f53a
-*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*-*-*-*-

FOR -A-
--------------------------------------------------------------------------------------------------
1. Install the netmap driver and the corresponding ixgbe-3.15.1/ driver
  - please go through netmap's documentation for installation
  instructions. We used the following command to set the compilation
  scripts for netmap (for ixgbe-3.15.1 driver).
  	   # ./configure --kernel-dir=/path/to/kernel/src --no-drivers=i40e,virtio_net.c
  - To run mTCP clients correctly, you need to modify the RSS
  seed in ixgbe_main.c:ixgbe_setup_mrqc() function. Our mTCP stack
  uses a specific RSS seed (mentioned below). 

   	   - seed[10] should be reset to { 
	     	      	  0x05050505, 0x05050505, 0x05050505,
			  0x05050505, 0x05050505, 0x05050505, 0x05050505,
			  0x05050505, 0x05050505, 0x05050505
	     };
  
  - Make sure that the underlying kernel module is correctly
  working. You can use sample applications to validate your 
  setup.
	   # make
	   # sudo insmod ./netmap.ko
	   # sudo insmod ./ixgbe/ixgbe.ko

FOR -B-
--------------------------------------------------------------------------------------------------
1. Install the netmap driver and the corresponding ixgbe-5.3.7/ driver
  - please go through netmap's documentaiton for installation
  instructions. We used the following command(s) to compile the driver
  	  # cd LINUX; ./configure; make

  - To run mTCP clients correctly, you need to modify the RSS seed in
  ixgbe_main.c:ixgbe_init_rss_key() function. Our mTCP stack uses a specific
  RSS seed (mentioned below).

      	   static inline int ixgbe_init_rss_key(struct ixgbe_adapter *adapter)
	   {
#if 0
		u32 *rss_key;
		if (!adapter->rss_key) {
			rss_key = kzalloc(IXGBE_RSS_KEY_SIZE, GFP_KERNEL);
			if (unlikely(!rss_key))
				return -ENOMEM;

			netdev_rss_key_fill(rss_key, IXGBE_RSS_KEY_SIZE);
			adapter->rss_key = rss_key;
		}
#endif
		static uint32_t sym_rss_key[10] = {
	     	      	  0x05050505, 0x05050505, 0x05050505,
			  0x05050505, 0x05050505, 0x05050505, 0x05050505,
			  0x05050505, 0x05050505, 0x05050505
		};

		if (!adapter->rss_key)
			adapter->rss_key = sym_rss_key;
		return 0;
	   }

  - Make sure that the underlying kernel module is correctly
  working. You can use sample applications to validate your 
  setup.
	   # make
	   # sudo insmod ./netmap.ko
	   # sudo insmod ./ixgbe/ixgbe.ko

For -C-
--------------------------------------------------------------------------------------------------
1. Install the netmap driver and the corresponding i40e-2.4.6/ driver
  - please go through netmap's documentation for installation
  instructions. We used the following command(s) to compile the driver
  	# cd LINIX; ./configure; make

  - To run mTCP clients correctly, you need to modify the RSS seed in
  i40e_main.c:i40e_config_rss_aq() & i40e_main.c:i40e_config_rss_reg()
  functions. Our mTCP stack uses a specific RSS seed (mentioned below).

  	     static unsigned char sym_seed[] = {
             	    0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
        	    0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
        	    0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
        	    0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
        	    0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
        	    0x05, 0x05
	     };

  	     static int i40e_config_rss_aq(struct i40e_vsi *vsi, const u8 *seed,
             	    	                   u8 *lut, u16 lut_size)
	     {
		struct i40e_pf *pf = vsi->back;
       		struct i40e_hw *hw = &pf->hw;
        	int ret = 0;

        	seed = sym_seed;

		...
	     }


	     static int i40e_config_rss_aq(struct i40e_vsi *vsi, const u8 *seed,
                              		   u8 *lut, u16 lut_size)
	     {
		struct i40e_pf *pf = vsi->back;
        	struct i40e_hw *hw = &pf->hw;
        	int ret = 0;

        	seed = sym_seed;

		...
	     }

  - Make sure that the underlying kernel module is correctly
  working. You can use sample applications to validate your
  setup.
	# make
	# sudo insmod ./netmap.ko
	# sudo insmod ./i40e/i40e.ko
	
FOR BOTH -A- & -B- & -C- (CONTINUED)
--------------------------------------------------------------------------------------------------
 2. For optimum performance you are suggested to bind NICS IRQs to arbitrary
    CPUs. Please use affinity-netmap.py script for this purpose. The current
    script is setup for the netmap ixgbe-5.3.7 driver. Please use a variant
    of this file for other cases (igb, i40e etc.).
  	   # ./config/affinity-netmap.py ${IFACE}

  - Disable flow control in Ethernet layer
    	   # sudo ethtool -A ${IFACE} rx off
    	   # sudo ethtool -A ${IFACE} tx off

  - Disable lro (large receive offload) in Ethernet device. mTCP
  does not support large packet sizes (> 1514B) yet)
       	   # sudo ethtool -K ${IFACE} lro off
	   # sudo ethtool -K ${IFACE} gro off	   

  - We used example/pktgen to test netmap raw network I/O
  performance. Netmap's pktgen can be used not only for
  packet generation but also packet reception. Since mTCP
  relies on RSS-based NIC hardware queues, we recommend using
  the following command-line arguments to test pkt-gen as a
  sink before testing mTCP for netmap.
  	   -SINK- (assuming the machine has 4 cpus)
	   # sudo ./pkt-gen -i ${IFACE}-0 -f rx -c 1 -a 0 -b 64 &
	   # sudo ./pkt-gen -i ${IFACE}-1 -f rx -c 1 -a 1 -b 64 &
	   # sudo ./pkt-gen -i ${IFACE}-2 -f rx -c 1 -a 2 -b 64 &
	   # sudo ./pkt-gen -i ${IFACE}-3 -f rx -c 1 -a 3 -b 64 &

  where ${IFACE} is netmap-enabled interface.
  The netmap README file shows a concise description on how to
  use the driver. We reiterate some points that are essential in
  understanding the command line arguments above. An interface
  name post-appended with a number means that the process will
  read traffic from the specified NIC hardware queue. `-a` argument
  lets the program bind to a specific core.
  
3. Setup mtcp library:
   	   # ./configure --enable-netmap
   	   # make
  - By default, mTCP assumes that there are 16 CPUs in your system.
    You can set the CPU limit, e.g. on a 32-core system, by using the following command:
    	   # ./configure --enable-netmap CFLAGS="-DMAX_CPUS=32"
    Please note that your NIC should support RSS queues equal to the MAX_CPUS value
    (since mTCP expects a one-to-one RSS queue to CPU binding).	   
  - In case `./configure' script prints an error, run the
    following command; and then re-do step-2 (configure again):
    	   # autoreconf -ivf
  - check libmtcp.a in mtcp/lib
  - check header files in mtcp/include
  - check example binary files in apps/example

4. Check the configurations in apps/example
  - epserver.conf for server-side configuration
  - epwget.conf for client-side configuration
  - you may write your own configuration file for your application

5. Run the applications. *If you run the application with one thread,
mTCP core will assume that the multi-queues option is disabled. This
assumption is only valid for netmap version.*

6. Netmap module (mtcp/src/netmap_module.c) by default uses blocking
I/O by default. Most microbenchmarking applications (epserver/epwget)
shows best performance with this setup in our testbed. In case the
performance is sub-optimal in yours, we recommend that you try polling
mode (by enabling CONST_POLLING in line 24). You can also try tweaking
IDLE_POLL_WAIT/IDLE_POLL_COUNT macros while testing blocking mode I/O.