CI/BUG: add native jobs for s390x, fix bug in `pack_inner` by AlekseiNikiforovIBM · Pull Request #30819 · numpy/numpy

AlekseiNikiforovIBM · 2026-02-11T14:55:39Z

Rename workflow linux-ppc64le.yml into linux-actionspz.yml.
Copy and update ppc64le workflow for s390x.
Disable qemu-based workflows for s390x.

Fix big endian issue in pack_inner.

Closes: #30817

@mkumatag @rgommers

mkumatag · 2026-02-11T15:14:16Z

@sandeepgupta12 ptal

AlekseiNikiforovIBM · 2026-02-11T16:39:22Z

So, currently clang tests are failing on ppc64le and s390x, and clang and gcc tests with forced vectorization fail on s390x.

Can we proceed with these workflows now or should tests be fixed first? I'm already looking into s390x failures.

AlekseiNikiforovIBM · 2026-02-11T17:38:26Z

This change fixes s390x tests with forced zvector:

663107d

I could also split it into a separate PR if you'd like.

Now there are only some failures when clang is used.

rgommers

Thanks @AlekseiNikiforovIBM. This is a bit tricky to review (let's blame the GitHub UI), so it'd be great if you could revert the name change and leave it at linux-ppc64le.yml for now. That will make the diff much smaller, and it'll be visible what you tweaked in the ppc64le job.

rgommers · 2026-02-11T20:19:16Z

.github/workflows/linux-ppc64le.yml

+      fail-fast: false
+      matrix:
+        config:
+          - name: "GCC (ppc64le)"


This whole job matrix is ppc64le, so this name change seem a bit unnecessary

It looks like it takes title of workflow file, and then job name. Without this change there will be two jobs named "Native ppc64le and s390x Linux Test / GCC", one for ppc64le and one for s390x.

rgommers · 2026-02-11T20:19:48Z

.github/workflows/linux-actionspz.yml

Good catch, export doesn't survive across job steps

rgommers · 2026-02-11T20:23:39Z

.github/workflows/linux-ppc64le.yml

+          - name: "GCC with zvector (s390x)"
+            args: "-Dallow-noblas=false -Dcpu-baseline=vx"
+          - name: "clang with zvector (s390x)"
+            args: "-Dallow-noblas=false -Dcpu-baseline=vx"


We had two jobs under QEMU; four seems a little excessive - can you do GCC and Clang with zvector here (or vice versa?

They produce different results, so I think it's better to have all 4 of them. Considering issues with clang, maybe clang jobs should be dropped until all issues when compiling with clang are resolved?

With clang 18, ppc64le tests are passing. See here- https://github.com/numpy/numpy/actions/runs/21948675141/job/63393053952

I don't see clang-18 being used there. I see gcc. From linked build log:

C compiler for the host machine: cc (gcc 13.3.0 "cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0") C linker for the host machine: cc ld.bfd 2.42 C++ compiler for the host machine: c++ (gcc 13.3.0 "c++ (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0") C++ linker for the host machine: c++ ld.bfd 2.42

Clang-20 logs for comparison:

C compiler for the host machine: clang-20 (clang 20.1.2 "Ubuntu clang version 20.1.2 (0ubuntu1~24.04.2)") C linker for the host machine: clang-20 ld.bfd 2.42 C++ compiler for the host machine: clang++-20 (clang 20.1.2 "Ubuntu clang version 20.1.2 (0ubuntu1~24.04.2)") C++ linker for the host machine: clang++-20 ld.bfd 2.42

rgommers · 2026-02-11T20:26:09Z

I could also split it into a separate PR if you'd like.

A single PR seems fine to me, thanks.

Add parameters for runner and compiler

Only 16-byte vectorization path is tested for big endian.

It contains fixes for numpy testsuite, compared to default clang-18

Too many tests still fail and need investigation.

AlekseiNikiforovIBM · 2026-02-12T15:44:00Z

I've disabled clang builds and tests for s390x while there are still too many issues to be fixed. I think enabling them later would be a good choice.

Newer clang from Fedora 43 builds numpy and passes tests better than clang from Ubuntu 24.04

AT_HWCAP is now a hex number and needs to be decoded. https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=e5893e6349541d871e8a25120bca014551d13ff5 Use /prc/cpuinfo to obtain same information.

Too many failing tests which need to be looked at.

AlekseiNikiforovIBM · 2026-02-13T15:12:01Z

I've updated PR.

I've used docker and fedora:43 image to build and test numpy with clang. Fedora has newer clang, and maybe clang is patched differently there, so there are no miscompilations detected by testsuite.

Only 1 test failed and was needed to be fixed: test_cpu_features.py::Test_ZARCH_Features::test_features.

AT_HWCAP is no longer decoded in glibc output:
https://sourceware.org/git/?p=glibc.git;a=commit;h=e5893e6349541d871e8a25120bca014551d13ff5
Now it's provided as a hex number:

$ LD_SHOW_AUXV=1 /bin/true | grep AT_HWCAP
AT_HWCAP:             0x67ffff

Luckily, on s390x a copy of output is available in /proc/cpuinfo and I've switched to using it:

$ grep feature /proc/cpuinfo 
features        : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx vxd vxe gs vxe2 vxp sort dflt pcimio sie

Unfortunately, I don't think similar output is available for ppc64le. Hex output of AT_HWCAP would be needed to be decoded for ppc64le when it'd be detected, or some other approach would have to be used.

I've also bumped zvector build to enable everything up to VXE2.

There still were 65 failing tests for ppc64le when building with clang:
https://github.com/numpy/numpy/actions/runs/21990958320/job/63537634918?pr=30819

2026-02-13T14:52:11.1751562Z = 65 failed, 47875 passed, 344 skipped, 2843 deselected, 34 xfailed, 3 xpassed in 256.65s (0:04:16) =

I disabled ppc64le clang build for now. It'd be better to have those tests fixed or disabled first, or some other build environment used.

@rgommers please take one more look.

AlekseiNikiforovIBM marked this pull request as draft February 11, 2026 15:05

AlekseiNikiforovIBM marked this pull request as ready for review February 11, 2026 16:39

rgommers reviewed Feb 11, 2026

View reviewed changes

rgommers added 03 - Maintenance component: CI component: SIMD Issues in SIMD (fast instruction sets) code or machinery labels Feb 11, 2026

rgommers requested a review from seiko2plus February 11, 2026 20:25

rgommers changed the title ~~Implement workflows for s390x native CI~~ CI/BUG: Implement workflows for s390x native CI Feb 12, 2026

rgommers changed the title ~~CI/BUG: Implement workflows for s390x native CI~~ CI/BUG: add native jobs for s390x, fix bug in pack_inner Feb 12, 2026

AlekseiNikiforovIBM force-pushed the s390x_ci branch from f3fa0c1 to 7f05a2d Compare February 12, 2026 11:42

AlekseiNikiforovIBM added 5 commits February 12, 2026 13:51

Implement workflow for s390x

2a1dd8e

Add parameters for runner and compiler

Disable qemu-based workflows for s390x

2eeff22

Fix test_packbits.py::test_packbits_large on s390x

0d07dba

Only 16-byte vectorization path is tested for big endian.

Fix setting CC and CXX for clang jobs

b8c3e34

Switch to clang-20

d3779d0

It contains fixes for numpy testsuite, compared to default clang-18

AlekseiNikiforovIBM force-pushed the s390x_ci branch from 7f05a2d to d3779d0 Compare February 12, 2026 12:52

Disable clang builds for s390x

8571eef

Too many tests still fail and need investigation.

AlekseiNikiforovIBM force-pushed the s390x_ci branch from 9bf91ef to 9dfb90e Compare February 13, 2026 13:27

AlekseiNikiforovIBM added 4 commits February 13, 2026 15:47

Implement clang CI for ppc64le and s390x using docker and fedora:43

662eb43

Newer clang from Fedora 43 builds numpy and passes tests better than clang from Ubuntu 24.04

Use /proc/cpuinfo to get CPU features on s390x

ded6310

AT_HWCAP is now a hex number and needs to be decoded. https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=e5893e6349541d871e8a25120bca014551d13ff5 Use /prc/cpuinfo to obtain same information.

Disable clang build for ppc64le

fb2e142

Too many failing tests which need to be looked at.

Bump zvector builds to VXE2

c9693ef

AlekseiNikiforovIBM force-pushed the s390x_ci branch from bae121c to c9693ef Compare February 13, 2026 14:52

Uh oh!

Conversation

AlekseiNikiforovIBM commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mkumatag commented Feb 11, 2026

Uh oh!

AlekseiNikiforovIBM commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlekseiNikiforovIBM commented Feb 11, 2026

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

rgommers Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

AlekseiNikiforovIBM Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

rgommers Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

rgommers Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

AlekseiNikiforovIBM Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

sandeepgupta12 Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

AlekseiNikiforovIBM Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rgommers commented Feb 11, 2026

Uh oh!

AlekseiNikiforovIBM commented Feb 12, 2026

Uh oh!

AlekseiNikiforovIBM commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AlekseiNikiforovIBM commented Feb 11, 2026 •

edited

Loading

AlekseiNikiforovIBM commented Feb 11, 2026 •

edited

Loading

AlekseiNikiforovIBM Feb 12, 2026 •

edited

Loading