Thanks Wil, from your link I've tried e1000, e1000e and vmxnet3, all with and without split RX mode (on a single VM) unfortunately without solving the problem. 
I've gone back to e1000 which I use everywhere else.
I've found two workarounds that will solve my immediate problem but I'm no closer to understanding *why* I've got the problem in the first place. To summarise the problem:
Multicast packets from a particular source device don't make it through into VM clients while packets from other sources do. See my original post for combinations I've tested.
As far as I can see, the test packets I am generating (which arrive everywhere) have the same characteristics as the packets generated by the electricity meter (which don't get into VMs). Here's a pair of packet headers captured from a host OSX machine:
TCPDUMP header of a failing packet:
13:18:42.745537 44:37:19:00:08:bc (oui Unknown) > 01:00:5e:c0:20:13 (oui Unknown), ethertype IPv4 (0x0800), length 368: (tos 0x0, ttl 64, id 4010, offset 0, flags [none], proto UDP (17), length 350)
192.168.201.83.22600 > 224.192.32.19.22600: [udp sum ok] UDP, length 322
TCPDUMP header of a working (test) packet:
13:18:43.092798 f0:de:f1:60:11:d9 (oui Unknown) > 01:00:5e:40:20:13 (oui Unknown), ethertype IPv4 (0x0800), length 629: (tos 0x0, ttl 64, id 25825, offset 0, flags [none], proto UDP (17), length 611)
ev016.evans6.local.22600 > 224.192.32.19.22600: [udp sum ok] UDP, length 583
(I've tried varying my test packet sizes from small to large but they always get through).
Now to the workarounds ...
libpcap
Following on from AKostur's observation, I've found that if my Java multicast listener program calls JpcapCaptor.openDevice(...) to open the interface in promiscuous mode (but without bothering to try reading from it) then I subsequently receive the previously dropped packets. When I make the call, fusion interactively prompts me to allow or deny the access but packets become visible immediately, regardless of whether I ignore the prompt, or answer yes or answer no. I've tested this with Win7 & Server 2008 guests.
I eventually want to run my listener in a headless Win 2003 server VM and I haven't tested it there yet but it looks promising.
Since jpcap is a wrapper for libpcap (or pcapwin in my test) then I expect the same effect could be obtained in C/C++ using libpcap directly.
re-cast (or re-tweet as I called it earlier)
I can receive all multicast packets in all physical machines, regardless of their origin and without consciously resorting to changing NIC characteristics. If I re-multicast the packets I receive, they subsequently arrive successfully everywhere else, including all VMs, without me having to set the VM NICs to promiscuous. This works the same whether I use C or Java to re-send.
Since the data volume of the multicasts is tiny (a couple per minute) then this duplication won't be a problem for me and it means I'll only need jpcap/libpcap in one place.
Moving On...
It's frustrating not to know why I'm seeing different behaviour for packets that are being sent from different devices but, having found a couple of workarounds, I'm not going to spend too much more time on it. A little bit more but not too much. I'll leave the thread open until I call it a day.
Just in case it's useful for anybody, here's my Java class that forces promiscuity on the interface and also the self-contained C program that re-sends the multicasts it receives.
package comdem.one;
import java.io.IOException;
import java.net.DatagramPacket;
import java.net.MulticastSocket;
import java.net.UnknownHostException;
import java.util.concurrent.ArrayBlockingQueue;
import jpcap.JpcapCaptor;
import jpcap.NetworkInterface;
import jpcap.NetworkInterfaceAddress;
/**
* Reads messages (as an array of bytes) from a multicast socket and makes
* each message available via a FIFO queue.
*/
public class MulticastReader
{
private final ArrayBlockingQueue <byte[]> packets;
private final MulticastReaderParameters mcrp;
public MulticastReader(MulticastReaderParameters mcrp) throws IOException
{
this.mcrp = mcrp;
this.packets = new ArrayBlockingQueue<byte[]>(100);
initialiseInterface();
}
private void initialiseInterface() throws IOException
{
makeInterfacePromiscuous(findInterfaceFromName());
}
private void makeInterfacePromiscuous(NetworkInterface ni)
throws IOException
{
JpcapCaptor dev =
JpcapCaptor.openDevice(ni, 2000, true, 0);
if(null == dev)
System.out.println("OPEN DEV failed");
}
private NetworkInterface findInterfaceFromName()
throws UnknownHostException
{
for(NetworkInterface ni: JpcapCaptor.getDeviceList())
{
for(NetworkInterfaceAddress nia: ni.addresses)
{
if (nia.address.equals(mcrp.getLocalInterfaceIP()))
{
return ni;
}
}
}
throw new UnknownHostException("" + mcrp.getLocalInterfaceIP());
}
public Thread start()
{
Thread t = new Thread()
{
public void run()
{
try
{
MulticastSocket s =
new MulticastSocket(mcrp.getMulticastPort());
s.joinGroup(mcrp.getMulticastIP());
byte[] buff = new byte[800];
System.out.println(
"Reading from port " +
mcrp.getMulticastIP() +
":" +
mcrp.getMulticastPort());
while(!interrupted())
{
DatagramPacket pkt =
new DatagramPacket(buff,buff.length);
s.receive(pkt);
enqueue(pkt.getData());
}
}
catch (Exception e)
{
e.printStackTrace();
System.out.println(
"Interrupted " +
Thread.currentThread().getName());
System.out.flush();
}
}
private void enqueue(byte[] packetPayload)
{
try
{
packets.add(packetPayload);
}
catch (IllegalStateException e)
{
System.err.println(
"queue full, discard: " + new String(packetPayload));
}
}
};
t.setName("Reader");
t.start();
return t;
}
public byte[] read() throws InterruptedException
{
return packets.take();
}
}
And the C ...
/*
* Joins a multicast group and echoes all data it receives from
* the group to another multicast port
*
* David Evans Nov-2012.
* Adapted from original by Antony Courtney & Fredric Bastien
*
*/
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <time.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#define MULTICAST_IP "224.192.32.19"
#define MULTICAST_PORT 22600
#define FORWARD_IP "224.192.32.19"
#define FORWARD_PORT 22602
#define MSGBUFSIZE 800
////////////////////////////////////////////////////////////////////////////////
typedef struct {
int fd;
struct sockaddr_in addr;
unsigned addr_len;
} PORT;
typedef struct {
char content[MSGBUFSIZE];
int length;
} MESSAGE;
////////////////////////////////////////////////////////////////////////////////
size_t rewrite(PORT destination, MESSAGE message);
PORT make_vanilla_port(char *multicast_IP, int multicast_port);
PORT make_multicast_listening_port(char *multicast_IP, int multicast_port);
void share_port_number(PORT port);
void bind_to_receive_address(PORT port);
void join_multicast_group(PORT port, char *multicast_IP);
MESSAGE read_message(PORT source);
////////////////////////////////////////////////////////////////////////////////
int main(int argc, char *argv[])
{
PORT source;
PORT destination;
printf(
"forwarding from %s:%d to %s:%d\n",
MULTICAST_IP,
MULTICAST_PORT,
FORWARD_IP,
FORWARD_PORT);
destination = make_vanilla_port(FORWARD_IP, FORWARD_PORT);
source = make_multicast_listening_port(MULTICAST_IP, MULTICAST_PORT);
while (1)
{
rewrite(
destination,
read_message(source));
}
}
////////////////////////////////////////////////////////////////////////////////
PORT make_vanilla_port(char *multicast_IP, int multicast_port)
{
PORT port;
if ((port.fd=socket(AF_INET,SOCK_DGRAM,0)) < 0)
{
perror("socket");
exit(1);
}
memset(&(port.addr),0,sizeof(port.addr));
port.addr.sin_family=AF_INET;
port.addr.sin_addr.s_addr=inet_addr(multicast_IP);
port.addr.sin_port=htons(multicast_port);
return port;
}
////////////////////////////////////////////////////////////////////////////////
PORT make_multicast_listening_port(char *multicast_IP, int multicast_port)
{
PORT port = make_vanilla_port(multicast_IP, multicast_port);
share_port_number(port);
bind_to_receive_address(port);
join_multicast_group(port,multicast_IP);
return port;
}
void share_port_number(PORT port)
{
u_int YES=1;
if (setsockopt(
port.fd,
SOL_SOCKET,
SO_REUSEADDR,
&YES,
sizeof(YES)) < 0)
{
perror("Reusing ADDR failed");
exit(1);
}
}
void bind_to_receive_address(PORT port)
{
if (bind(
port.fd,
(struct sockaddr *) &(port.addr),
sizeof(port.addr)) < 0)
{
perror("bind");
exit(1);
}
}
void join_multicast_group(PORT port, char *multicast_IP)
{
struct ip_mreq mreq;
mreq.imr_multiaddr.s_addr=inet_addr(multicast_IP);
mreq.imr_interface.s_addr=htonl(INADDR_ANY);
if (setsockopt(
port.fd,
IPPROTO_IP,
IP_ADD_MEMBERSHIP,
&mreq,
sizeof(mreq)) < 0)
{
perror("setsockopt");
exit(1);
}
}
////////////////////////////////////////////////////////////////////////////////
MESSAGE read_message(PORT source)
{
MESSAGE message;
if (0 >= (message.length =
(int)recvfrom(
source.fd,
message.content,
MSGBUFSIZE,
0,
(struct sockaddr *) &(source.addr),
&(source.addr_len))))
{
perror("recvfrom");
exit(1);
}
return message;
}
////////////////////////////////////////////////////////////////////////////////
size_t rewrite(PORT destination, MESSAGE message)
{
puts(message.content);
return sendto(
destination.fd,
message.content,
message.length,
0,
(struct sockaddr *) &(destination.addr),
sizeof(destination.addr));
}
////////////////////////////////////////////////////////////////////////////////