> I need to figure out if PLX switching is required,
> or if the lanes can simply be separated via routing.

There's some info on PCIe bifurcation/trifurcation on the Talos II (Lite):

https://www.phoronix.com/forums/forum/hardware/processors-memory/1025053-raptor-launching-talos-ii-lite-power9-computer-system-at-a-lower-cost?p=1025214#post1025214
https://www.phoronix.com/forums/forum/hardware/processors-memory/1025053-raptor-launching-talos-ii-lite-power9-computer-system-at-a-lower-cost?p=1025310#post1025310
https://www.phoronix.com/forums/forum/hardware/processors-memory/1025053-raptor-launching-talos-ii-lite-power9-computer-system-at-a-lower-cost?p=1025326#post1025326
http://tenfourfox.blogspot.com/2018/05/a-little-talos-of-your-very-own.html

You da man, Andreas.

I need this kind of stuff.

BTW, anything similar for the NXPT4240?
In particular, can the SerDes lanes assigned to network connectivity be assigned to other devices?

Quote:

Well, the summary of that seems to state that the X8 slot could be bifurcated but the primary reason its problematic is that those lanes also control another device.
I could use some information on that device and which lanes it is connect to (some or all).

In any case, as the Talos II Lite has both USB 2.0 and 3.0 connections,sound via these is an option, as is sound from a Radeon HD HDMI output.
What other devices do we need?
I can run Linux from that combination without even using the X8 slot.

[ Edited by Jim 21.05.2018 - 11:45 ]

> anything similar for the NXPT4240? [...] can the SerDes lanes assigned to
> network connectivity be assigned to other devices?

Except by selecting another predefined SerDes config that assigns less SerDes lanes to the network controllers and more to other devices, no.

> the summary of that seems to state that the X8 slot could be bifurcated
> but the primary reason its problematic is that those lanes also control
> another device. I could use some information on that device and which
> lanes it is connect to (some or all).

As I understand it, the x8 slot on the Talos II Lite comes from an x16 controller that already bifurcates into this x8 slot and a second x8 connection to the SAS controller. The x8+x8 bifurcation of this x16 controller is fixed in silicon and thus cannot be changed.

Quote:

Not even using a PLX switch?
And the SAS controller is optional, the board appears to be available without it.

>The x8+x8 bifurcation of this x16 controller is fixed in silicon and thus cannot be changed.<

That is also how I understood the PCIe breakout from the CPU, hence the dual cpu board is what I ordered.

I know it's more expensive, but for the future, perhaps a single cpu on a dual board is a better buy.

>> The x8+x8 bifurcation of this x16 controller is fixed in silicon
>> and thus cannot be changed.

> Not even using a PLX switch?

I think using a PCIe switch on the x8 slot could work as this has nothing to do with the bifurcation of the POWER9's PCIe controller.

> the SAS controller is optional, the board appears to be available without it.

I guess the lanes dedicated to this SAS controller are routed to the footprint no matter if populated or not.

Quote:

Yeah, I'm trying to figure out the available configurations.
There appears to be a way to configure more PCI-E lane than the X8, X4 combo used on the T4240RDB-PB (at least 16 lanes, possibly 17), but I'm not sure that that doesn't disable SATA.
One I2C lane, I2C4, isn't even used on that reference board, and other SerDes lane groups appear not to be fully utilized.
It going to take a little more examination of the SOC vs the board documentation to sort this out.

> There appears to be a way to configure more PCI-E lane than the X8,
> X4 combo used on the T4240RDB-PB (at least 16 lanes, possibly 17)

16 is the maximum. I already researched the SerDes configurations of the T4240/T4241 in 2013, and again last year:

https://morph.zone/modules/newbb_plus/viewtopic.php?forum=11&topic_id=7901&start=49
https://morph.zone/modules/newbb_plus/viewtopic.php?forum=11&topic_id=12151&start=39

> I'm not sure that that doesn't disable SATA.

Yes, 16 PCIe lanes means disabled SATA. The maximum PCIe lanes with SATA enabled is 14 (see links above).

> other SerDes lane groups appear not to be fully utilized.

Yes, T4240QDS shows better utilization of the SoC's capabilities than T4240RDB.

Andreas, you are the man, your recall saves me so much time.

Thanks for those two references.

So,if available configurations include

1. x8-gen2 x8-gen2
2. x8-gen2 x4-gen3 x4-gen2
3. x4-gen3 x4-gen2 x4-gen2 x4-gen2

Wouldn't option 3 make sense since an X4 gen 3 slot has the same bandwith as an X8 gen 2 slot?

And that would mean that the T4240QDS likely uses this last option?
If the T4240RDB-PB uses something similar to option 2, is it utilizing gen3 lanes on the single X4 slot, and what about those two remaining PCI-e lanes even if SATA is retained?

What does that leave?

X8 gen2, X4 gen3 (or2) with two lanes unused.
Or, X4 gen3, X4 gen2, X4 gen2 again with two lanes unused (and two reserved for SATA 2).

Seems like not utilizing the onboard SATA controller and going with an X4 gen3 (to supply an X16 slot) for the video card, with two X4 gen2 slots and four lanes serving an SATAIII controller would make more sense.

Three slots, instead of two, and the same bandwidth for video.

[ Edited by Jim 22.05.2018 - 17:06 ]

I just got a reply from Raptor Engineering.
One of the things I asked about was splitting the Talos II Lite's x8 PCI-E slot.

So:

>Has anyone attempted to use an expansion system (plx based or some other solution) that would break that X8 slot into multiple slots with fewer PCI-E lanes.

"Yes! We actually have several of these systems under test. If you're willing to work out the custom mounting arrangements, we can provide a list of hardware that could break that x8 slot to either two or four x4 Gen 3 slots"

Note, that mentions two x4 or four x4 Gen 3 slots.
I'm not sure if that means two x4 Gen 4 slots or four x4 Gen 3 slots, or if all options are Gen 3.

They did confirm that they are working on a reduced form factor board.

> Note, that mentions two x4 or four x4 Gen 3 slots.

That's probably because Raptor so far only tested PCIe gen3 bridge/switch/split cards (if there are even PCIe gen4 ones out already).

> I'm not sure if that means two x4 Gen 4 slots or four x4 Gen 3 slots,
> or if all options are Gen 3.

I guess they mean the latter. Bridging an x8 gen4 slot into two x4 gen4 slots should be possible as well, of course, but someone would have to actually test such card with the board.

>> I'm not sure if that means two x4 Gen 4 slots or four x4 Gen 3 slots,
>> or if all options are Gen 3.

>I guess they mean the latter. Bridging an x8 gen4 slot into two x4 gen4 slots should be possible as well, >of course, but someone would have to actually test such card with the board.

That does seem likely, and the comment about four x4 slots probably doesn't mean four x4 slots that have all lanes connected (unless they are talking about a switch).

But its beginning to look like a simple x8 to dual x4 split might be possible.

Addendum:

> I don't think any of this makes a difference when emulating/virtualizing
> the systems with QEMU/KVM.

It seems using QEMU/KVM to virtualize (as opposed to emulate) Power ISA CPUs (except POWER8 and POWER9) is not possible on the Talos II:

"On to QEMU. Using the generic Power Mac profile mac99 both MacOS 9.1 and 10.4 start up largely happily under qemu-system-ppc [...]. However, while it was certainly useable, it didn't feel very fast. The System Profiler within the emulated Tiger instance said it was a "1GHz G4" with a "400MHz FSB." This seemed low, and the reason it is was ... drumroll please ... it was running with CPU emulation. After some checking, I confirmed KVM was indeed installed on this system, so I tried running a 64-bit guest with qemu-system-ppc64 emulating an IBM pSeries machine with KVM-HV. That started up and ran at a nice clip, noticeably faster when I turned on KVM, so I tried to run the 32-bit guest with KVM-PR (which ought to emulate the proper CPU) and got an error message. Even the 64-bit guest that ran just dandy with KVM-HV wouldn't run with KVM-PR. Some digging determined that the KVM-PR kernel module existed, but did not load. Some more digging turned up that KVM-PR wouldn't load with modprobe. Even more digging turned up that ... KVM-PR doesn't run on bare-metal POWER9 yet, and unfortunately all PowerNV machines like the T2 are bare-metal."
http://tenfourfox.blogspot.com/2018/05/a-weekend-on-new-computer-or.html

"QEMU's performance as a Power Mac emulator is currently acceptable on the T2, just unspectacular.

Unspectacular? 1 GHz G4 level 32 bit performance ought to be considered quite underwhelming.

Sounds like 64 bit performance is a little better, but neither of the emulated cpus run at their full potential.

BTW - Thanks for the reference. I now more fully understand what Raptor mentioned in the disconnect between KVM and QEMU.
Now if only they can get this running as a 'bare metal' application.

You would think that 32 and 64 bit sessions would be capable of being emulated at close to full speed.
With a 16 thread processor, that would mean 16 potential G4 or G5 level sessions all running concurrently.

Remember what I said in the past about a cross-session hypervisor enabled update of Ambient that could composite multiple sessions?
By developing something like that we would start with our standard OS, move to a multi-session ASMP variation, and develop an SMP version that could run on as many cores as were assigned to it.
And later, run any combinations we wanted, possibly even with OSX, or OS4 sessions all at the same time.

> Sounds like 64 bit performance is a little better

In CPU emulation? It's near native performance when virtualizing a POWER9 or POWER8 system with KVM-HV.

> but neither of the emulated cpus run at their full potential.

Yes, that's the nature of CPU emulation. KVM-PR is needed to run virtualized Power ISA CPUs other than POWER(N) or POWER(N-1) to the host CPU's full potential on a POWER(N) system.

> You would think that 32 and 64 bit sessions would be capable of being emulated
> at close to full speed.

...only if virtualized with KVM instead of emulated :-)

> With a 16 thread processor, that would mean 16 potential G4 or G5 level sessions
> all running concurrently.

...as soon as KVM-PR works on PowerNV, if the session is wished to be faster than 1 GHz G4.

>...as soon as KVM-PR works on PowerNV, if the session is wished to be faster than 1 GHz G4.

That was what I got out of the article you provided pointers too, thanks.
Its reassuring.

> 1 GHz G4 level 32 bit performance ought to be considered quite underwhelming.

Progress has been made. KVM-PR hacked to run and virtualize a 2 GHz G4:

http://tenfourfox.blogspot.com/2018/06/another-weekend-on-new-computer-or.html

> new Talos II Single-CPU Board! Link
> 1780 $ for a complete system w. CPU (wo. RAM)

Talos II Lite mainboard available for 1100 USD:

https://www.raptorcs.com/TALOSIILITE/
https://www.raptorcs.com/content/TL1MB1/intro.html


With a quad-core POWER9 for 375 USD, this amounts to 1475 USD for board with CPU.

Came across this and some may find it interesting on the T2Lite.(?)

VideoCard x8/x16 shootout

Quote:

Talks II Lite use pci-e 4.0 slot, those benches you are referencing are for a PCI-E 3.0 test platform.
But yes, you could put the video card in the PCI-E x8 slot, and if the x16 slot could be bifurcated, you could have an x8, x4, and four x1 slots.

> Talks II Lite use pci-e 4.0 slot, those benches you are referencing
> are for a PCI-E 3.0 test platform.

As long as there are no PCIe v4 graphics cards, PCIe v3 or lower graphics cards will have to be used anyway, and the PCIe v4 spec of the Talos II will bring no advantage over the lowest common denominator which is the PCIe version of the graphics card.

> if the x16 slot could be bifurcated, you could have an x8, x4, and four x1 slots.

This would be more like hexafurcating ;-)

That is true.

I am no fan of x86 machines so a lot of stuff I read about it pretty much goes into garbage collection. However, I do recall something about the way the multicore processors are connected can become a bottleneck under heavy load and effect graphics performance due to the PCIe bus not getting the data in a timely fashion.

For me it does not really matter. I don't play graphics intensive games and these old eyes cannot be improved with higher resolutions.

Sometimes more fluff is exactly that, more fluff.

Hexafurcating. Sounds like something sexual that could get you arrested.

Hexafurcating?
Well at least that is amusing (its been a bad weekend).

I should have asked Raptor about that idea when I inquired about splitting the x8 slot.

That option would make the Talos II Lite more attractive.